Re: [PATCH] doc: Add technical overview

[Alexander Smirnov <asmirnov@ilbers.de>]

On 08/08/2017 05:53 PM, Henning Schild wrote:
> Am Tue, 8 Aug 2017 17:44:30 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> On 08/08/2017 05:38 PM, Henning Schild wrote:
>>> Am Tue, 8 Aug 2017 16:12:02 +0300
>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>    
>>>> On 08/08/2017 04:03 PM, Henning Schild wrote:
>>>>> Am Tue, 8 Aug 2017 15:41:10 +0300
>>>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>>>       
>>>>>> On 08/08/2017 02:54 PM, Henning Schild wrote:
>>>>>>> Am Tue, 8 Aug 2017 14:32:00 +0300
>>>>>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>>>>>          
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> On 08/08/2017 02:11 PM, Henning Schild wrote:
>>>>>>>>> I did not really read this yet. At the moment i think we have
>>>>>>>>> too many changes in the q to add such a document to the
>>>>>>>>> repository. Any change would just mean heaving to keep it up
>>>>>>>>> to date. We talked about the need for such a document for the
>>>>>>>>> possible OE integration. I think it is valuable for that
>>>>>>>>> already.
>>>>>>>>
>>>>>>>> Briefly speaking, this document describes initial motivation
>>>>>>>> and high-level current overview.
>>>>>>>>         
>>>>>>>>> That should be added after we all are happy with the
>>>>>>>>> feature-set of Isar, until then it would just be another file
>>>>>>>>> to update when changing how things work.
>>>>>>>>
>>>>>>>> Hmm, this document doesn't affect any discussions about Isar
>>>>>>>> feature set. It's intended to be as technical background for
>>>>>>>> hot-plug joining to this project. During discussions last
>>>>>>>> month I see that we have gap in our understandings how Isar
>>>>>>>> works, so this document should be start point to eliminate
>>>>>>>> such gaps.
>>>>>>>>
>>>>>>>> For sure, it's *not* a specification, which explicitly defines
>>>>>>>> specific feature set and nothing more will be added. It's just
>>>>>>>> an explanation of current state. I see no problems to regularly
>>>>>>>> update it, moreover I highly appreciate this. Lack of
>>>>>>>> documentation is the most common problem of open source
>>>>>>>> projects, so as soon we start documenting the things, the
>>>>>>>> easier will be documentation maintenance in future.
>>>>>>>
>>>>>>> For sure we need that! But especially section 3 contains lots of
>>>>>>> interna that will change if my patches get accepted. That is
>>>>>>> why i would like to postpone adding the file after my patches
>>>>>>> went in.
>>>>>>
>>>>>> As I said, this document describes current project state, so I
>>>>>> don't think that upcoming changes should block publishing of
>>>>>> documentation.
>>>>>>
>>>>>>      
>>>>>>>
>>>>>>> Adding that file before them would mean another rebase round for
>>>>>>> documentation. I would be happy to rebase this one for you on
>>>>>>> top of a master that contains my stuff, but not too happy about
>>>>>>> the other way around.
>>>>>>
>>>>>> The patch creates document from the scratch, so it can't conflict
>>>>>> with any upcoming patches.
>>>>>
>>>>> It conflicts with other patches that are under review right now.
>>>>> All i am asking is to merge conflicting patches in the order the
>>>>> patches initially showed up for review, in that case this one
>>>>> comes last anyways. And when its time has come it will be
>>>>> outdated, and i will update it for you.
>>>>>       
>>>>
>>>> Hmm, are you speaking about exactly this patch or about previous
>>>> series?
>>>
>>> Section 3.4 and 3.5 conflict with my changes, they will have to be
>>> changed. If your patch comes first i would have to rebase.
>>>
>>> i.e.
>>> - 3.4 talks about dpkg.bbclass which i renamed
>>> [PATCH 11-16 of 16 v2 4/6]
>>> - 3.5 will not happen anymore if everything enters the rootfs
>>> through multistrap
>>> [PATCH 3/3] classes: image: remove populate and replace it with a
>>> custom repo
>>
>> I didn't get the idea, how this affects your patches? After your
>> features will be applied, you can update this document. In this
>> thread you've mentioned, that this patch breaks your series and force
>> you to perform rebase, this point is not clear for me.
> 
> A consistent patch updates the code along with the documentation. If
> the code i am touching suddenly gets documented my commits will become
> inconsistent.

You can update the documentation later after your series will be applied.

Alex

>>>
>>> Henning
>>>
>>>    
>>>> Alex
>>>>   
>>>>>> Also I expect, that the author of new features will update the
>>>>>> documentation, so some reference document should be published
>>>>>> anyway.
>>>>>
>>>>> Sure.
>>>>>       
>>>>>> To be honest I'm not happy with the idea that I'll document all
>>>>>> the new features in future.
>>>>>
>>>>> Missing or inconsistent documentation are a valid reason to not
>>>>> accept a patch so it will not come to that.
>>>>>
>>>>> Henning
>>>>>       
>>>>>> Alex
>>>>>>      
>>>>>>>>>
>>>>>>>>> Am Tue, 8 Aug 2017 13:05:59 +0300
>>>>>>>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>>>>>>>             
>>>>>>>>>> Add initial Isar technical overview.
>>>>>>>>>>
>>>>>>>>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>>>>>>>>>> ---
>>>>>>>>>>       doc/technical_overview.md | 110
>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++ 1 file
>>>>>>>>>> changed, 110 insertions(+) create mode 100644
>>>>>>>>>> doc/technical_overview.md
>>>>>>>>>>
>>>>>>>>>> diff --git a/doc/technical_overview.md
>>>>>>>>>> b/doc/technical_overview.md new file mode 100644
>>>>>>>>>> index 0000000..f65fc08
>>>>>>>>>> --- /dev/null
>>>>>>>>>> +++ b/doc/technical_overview.md
>>>>>>>>>> @@ -0,0 +1,110 @@
>>>>>>>>>> +# 1 Introduction
>>>>>>>>>> +
>>>>>>>>>> +Isar can be shortly described as a set of bitbake recipes
>>>>>>>>>> that implement main build system logic. To simplify overall
>>>>>>>>>> Isar understanding, this document is split into two main
>>>>>>>>>> parts:
>>>>>>>>>> + - Isar logical components
>>>>>>>>>> + - Isar internal processes
>>>>>>>>>> +
>>>>>>>>>> +# 2 Isar Logical Components
>>>>>>>>>> +
>>>>>>>>>> +In this chapter the most important Isar components are
>>>>>>>>>> considered in details. In this text component doesn't
>>>>>>>>>> especially mean some self-contained single entity of
>>>>>>>>>> something, it's just an attempt to split Isar internals by
>>>>>>>>>> various criteria. + +### 2.1 Bitbake and Recipes + +All the
>>>>>>>>>> processes in Isar are started by bitbake, so it manages
>>>>>>>>>> general build executing process. Recipes in Isar can be split
>>>>>>>>>> in two categories: +
>>>>>>>>>> + - System recipes and classes: they are responsible for
>>>>>>>>>> setting up Debian-like infrastructure, manages Debian tools
>>>>>>>>>> execution for package building and installation, generation
>>>>>>>>>> root file system images
>>>>>>>>>> + - User recipes: custom user applications, that should be
>>>>>>>>>> builе from sources +
>>>>>>>>>> +There are two types of dependencies in Isar:
>>>>>>>>>> + - Dependency between bitbake recipes
>>>>>>>>>> + - Dependencies in Debian filesystem
>>>>>>>>>> +
>>>>>>>>>> +**NOTE:** Isar doesn't manage dependencies in Debian file
>>>>>>>>>> systems. User can only list specific Debian dependencies in
>>>>>>>>>> recipe, so they eventually will be passed to apt-get or
>>>>>>>>>> multistrap. Dependency installation is managed by Debian
>>>>>>>>>> tools.
>>>>>>>>>> + +### 2.2 Stamps + +Each task managed by bitbake uses stamp
>>>>>>>>>> to notify that it has been completed. Due to Isar supports
>>>>>>>>>> various Debian distributions and parallel builds for
>>>>>>>>>> multiple machines and architectures, the stamps for tasks
>>>>>>>>>> use special suffixes that include:
>>>>>>>>>> + - Debian distro name
>>>>>>>>>> + - Architecture
>>>>>>>>>> + - Machine
>>>>>>>>>> +
>>>>>>>>>> +Typical example, when Isar builds the following
>>>>>>>>>> configurations:
>>>>>>>>>> + - Debian Jessie, amd64
>>>>>>>>>> + - Debian Jessie, i386
>>>>>>>>>> + - Debian Stretch, i386
>>>>>>>>>> +
>>>>>>>>>> +In this case there will be 3 different buildchroots, so
>>>>>>>>>> standard hello demo application should be processed 3 times
>>>>>>>>>> for each environment. Three different sets of stamps should
>>>>>>>>>> be used for correct bitbake operating. + +### 2.3
>>>>>>>>>> Buildchroot + +One of the key aspect of Debian philosophy
>>>>>>>>>> claims the fact, that everything in Debian should be built
>>>>>>>>>> within Debian environment. Moreover native compilation is
>>>>>>>>>> more preferable than cross-compilation. To follow this
>>>>>>>>>> rules, Isar introduces the new component - buildchroot.
>>>>>>>>>> Bulidchroot is typical Debian filesystem that is created
>>>>>>>>>> using standard Debian tools: multistrap, apt. The source of
>>>>>>>>>> packages can be either official Debian repositories or
>>>>>>>>>> custom repositories created by user. + +Buildchroot
>>>>>>>>>> lifecycle can be described as following:
>>>>>>>>>> + - Buildchroot has initial configuration file which is
>>>>>>>>>> passed to multistrap tool. This configuration file is
>>>>>>>>>> generated by bitbake recipe from patterns and values defined
>>>>>>>>>> by user. Based on this configuration file, multistrap
>>>>>>>>>> generates initial filesystem.
>>>>>>>>>> + - During building custom Debian package, list of its build
>>>>>>>>>> dependencies is installed to buildchroot.
>>>>>>>>>> + - When package has been built, it's installed to current
>>>>>>>>>> buildchroot to satisfy further packages build dependencies. +
>>>>>>>>>> +### 2.4 Target Root Filesystem
>>>>>>>>>> +
>>>>>>>>>> +Target filesystem is quite similar to buildchroot. The only
>>>>>>>>>> difference is that it doesn't have development packages
>>>>>>>>>> installed.
>>>>>>>>>> + +Target filesystem lifecycle can be described as following:
>>>>>>>>>> + - Target filesystem has initial configuration file which is
>>>>>>>>>> passed to multistrap tool. This configuration file is
>>>>>>>>>> generated by bitbake recipe from patterns and values defined
>>>>>>>>>> by user. Based on this configuration file, multistrap
>>>>>>>>>> generates initial filesystem.
>>>>>>>>>> + - According to the list of custom packages in bitbake
>>>>>>>>>> recipes, the initial filesystem will be populated by
>>>>>>>>>> successfully built packages. + +# 3 Isar Internal Processes
>>>>>>>>>> +
>>>>>>>>>> +### 3.1 General Overview
>>>>>>>>>> +
>>>>>>>>>> +Whole Isar build process can be split into the following
>>>>>>>>>> steps:
>>>>>>>>>> + - Generation of initial buildchroots for each configuration
>>>>>>>>>> (Debian distro, machine and architecture) requested by user.
>>>>>>>>>> + - Generation of initial target filesystems for each
>>>>>>>>>> configuration.
>>>>>>>>>> + - Building custom packages.
>>>>>>>>>> + - Populate target filesystems.
>>>>>>>>>> + - Generate bootable images.
>>>>>>>>>> +
>>>>>>>>>> +All these steps are described in details below.
>>>>>>>>>> +
>>>>>>>>>> +### 3.2 Initial Buildchroot Generation
>>>>>>>>>> +
>>>>>>>>>> +As mentioned above, initial buildchroot is generated using
>>>>>>>>>> multistrap. The bitbake recipe which is responsible for
>>>>>>>>>> buildchroot can be found here:
>>>>>>>>>> `meta/recipes-devtools/buildchroot/buildchroot.bb`
>>>>>>>>>> + +This recipe implementes `do_build` task which performs the
>>>>>>>>>> following: +1. Generates multistrap config from template:
>>>>>>>>>> `meta/recipes-devtools/buildchroot/files/multistrap.conf.in`
>>>>>>>>>> +2. Install pre/post scripts for multistrap:
>>>>>>>>>> `meta/recipes-devtools/buildchroot/files/configscript.sh` and
>>>>>>>>>> `meta/recipes-devtools/buildchroot/files/setup.sh` +3. Run
>>>>>>>>>> multistrap +4. Install script for building custom Debian
>>>>>>>>>> packages: `meta/recipes-devtools/buildchroot/files/build.sh`
>>>>>>>>>> + +The single stamp is created for each user buildchroot
>>>>>>>>>> configuration. + +### 3.3 Initial Target Filesystem
>>>>>>>>>> Generation
>>>>>>>>>> + +Initial target filesystem generation process is very
>>>>>>>>>> similar to buildchroot creating, the difference is only in
>>>>>>>>>> initial packages list. + +Target image recipes are the part
>>>>>>>>>> of Isar core. There is a sample of typical Isar image that
>>>>>>>>>> can be customized according to the user requirements:
>>>>>>>>>> `meta-isar/recipes-core/images/isar-image-base.bb` +Like for
>>>>>>>>>> buildchroot, the multistrap configuration files for image can
>>>>>>>>>> be found here: `meta-isar/recipes-core/images/files`, and it
>>>>>>>>>> implements `do_build` task. + +### 3.4 Building Custom
>>>>>>>>>> Packages
>>>>>>>>>> + +Isar provides possibility to build Debian packages from
>>>>>>>>>> sources. This features works with Debian-like source
>>>>>>>>>> packages, i.e. the source code tree should contain debian
>>>>>>>>>> folder. The build process is implemented in
>>>>>>>>>> `meta/classes/dpkg.bbclass` and consists from the following
>>>>>>>>>> steps: +1. Task `do_fetch`: fetch source code from external
>>>>>>>>>> link +2. Task `do_unpack`: unpack source code to
>>>>>>>>>> `${BUILDCHROOT_DIR}/home/build/${PN}` +3. Task `do_build`:
>>>>>>>>>> switch to buildchroot using chroot command and run
>>>>>>>>>> `build.sh` script. The `build.sh` script performs the
>>>>>>>>>> following:
>>>>>>>>>> +   1. Go to `/home/build/${PN}`
>>>>>>>>>> +   2. Get list of dependencies from debian/control and
>>>>>>>>>> install them using apt.
>>>>>>>>>> +   3. Run dpkg-buildpackage
>>>>>>>>>> +4. Task `do_install`: install successfully built packages
>>>>>>>>>> `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy
>>>>>>>>>> directory `${DEPLOY_DIR_DEB}` + +### 3.5 Populate Target
>>>>>>>>>> Filesystem +
>>>>>>>>>> +Each target image can be extended by custom packages listed
>>>>>>>>>> in IMAGE_INSTALL variable. Task `do_populate` performs the
>>>>>>>>>> following: +1. Parse IMAGE_INSTALL variable +2. Find
>>>>>>>>>> respective packages in `${DEPLOY_DIR_DEB}` +3. Copy them to
>>>>>>>>>> deb folder in dedicated target filesystem +4. Execute dpkg
>>>>>>>>>> command in chroot for all the copied packages +
>>>>>>>>>> +### 3.6 Generate Bootable Image
>>>>>>>>>> +
>>>>>>>>>> +This process contains the following steps:
>>>>>>>>>> +1. Task `do_ext4_image`: target filesystem is packed to
>>>>>>>>>> extfs image +2. wic tool generates bootable image for
>>>>>>>>>> dedicated platform
>>>>>>>>>             
>>>>>>>          
>>>>>>      
>>>>>       
>>>>   
>>>    
>>
> 

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov