Re: [PATCH] doc: Add technical overview

[Alexander Smirnov <asmirnov@ilbers.de>]

On 08/08/2017 04:03 PM, Henning Schild wrote:
> Am Tue, 8 Aug 2017 15:41:10 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> On 08/08/2017 02:54 PM, Henning Schild wrote:
>>> Am Tue, 8 Aug 2017 14:32:00 +0300
>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>    
>>>> Hi,
>>>>
>>>> On 08/08/2017 02:11 PM, Henning Schild wrote:
>>>>> I did not really read this yet. At the moment i think we have too
>>>>> many changes in the q to add such a document to the repository.
>>>>> Any change would just mean heaving to keep it up to date.
>>>>> We talked about the need for such a document for the possible OE
>>>>> integration. I think it is valuable for that already.
>>>>>       
>>>>
>>>> Briefly speaking, this document describes initial motivation and
>>>> high-level current overview.
>>>>   
>>>>> That should be added after we all are happy with the feature-set
>>>>> of Isar, until then it would just be another file to update when
>>>>> changing how things work.
>>>>
>>>> Hmm, this document doesn't affect any discussions about Isar
>>>> feature set. It's intended to be as technical background for
>>>> hot-plug joining to this project. During discussions last month I
>>>> see that we have gap in our understandings how Isar works, so this
>>>> document should be start point to eliminate such gaps.
>>>>
>>>> For sure, it's *not* a specification, which explicitly defines
>>>> specific feature set and nothing more will be added. It's just an
>>>> explanation of current state. I see no problems to regularly update
>>>> it, moreover I highly appreciate this. Lack of documentation is the
>>>> most common problem of open source projects, so as soon we start
>>>> documenting the things, the easier will be documentation
>>>> maintenance in future.
>>>
>>> For sure we need that! But especially section 3 contains lots of
>>> interna that will change if my patches get accepted. That is why i
>>> would like to postpone adding the file after my patches went in.
>>
>> As I said, this document describes current project state, so I don't
>> think that upcoming changes should block publishing of documentation.
>>
>>
>>>
>>> Adding that file before them would mean another rebase round for
>>> documentation. I would be happy to rebase this one for you on top
>>> of a master that contains my stuff, but not too happy about the
>>> other way around.
>>
>> The patch creates document from the scratch, so it can't conflict
>> with any upcoming patches.
> 
> It conflicts with other patches that are under review right now. All i
> am asking is to merge conflicting patches in the order the patches
> initially showed up for review, in that case this one comes last
> anyways. And when its time has come it will be outdated, and i will
> update it for you.
> 

Hmm, are you speaking about exactly this patch or about previous series?

Alex

>> Also I expect, that the author of new features will update the
>> documentation, so some reference document should be published anyway.
> 
> Sure.
> 
>> To be honest I'm not happy with the idea that I'll document all the
>> new features in future.
> 
> Missing or inconsistent documentation are a valid reason to not accept
> a patch so it will not come to that.
> 
> Henning
> 
>> Alex
>>
>>>>>
>>>>> Am Tue, 8 Aug 2017 13:05:59 +0300
>>>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>>>       
>>>>>> Add initial Isar technical overview.
>>>>>>
>>>>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>>>>>> ---
>>>>>>     doc/technical_overview.md | 110
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed,
>>>>>> 110 insertions(+) create mode 100644 doc/technical_overview.md
>>>>>>
>>>>>> diff --git a/doc/technical_overview.md
>>>>>> b/doc/technical_overview.md new file mode 100644
>>>>>> index 0000000..f65fc08
>>>>>> --- /dev/null
>>>>>> +++ b/doc/technical_overview.md
>>>>>> @@ -0,0 +1,110 @@
>>>>>> +# 1 Introduction
>>>>>> +
>>>>>> +Isar can be shortly described as a set of bitbake recipes that
>>>>>> implement main build system logic. To simplify overall Isar
>>>>>> understanding, this document is split into two main parts:
>>>>>> + - Isar logical components
>>>>>> + - Isar internal processes
>>>>>> +
>>>>>> +# 2 Isar Logical Components
>>>>>> +
>>>>>> +In this chapter the most important Isar components are
>>>>>> considered in details. In this text component doesn't especially
>>>>>> mean some self-contained single entity of something, it's just
>>>>>> an attempt to split Isar internals by various criteria. + +###
>>>>>> 2.1 Bitbake and Recipes + +All the processes in Isar are started
>>>>>> by bitbake, so it manages general build executing process.
>>>>>> Recipes in Isar can be split in two categories: +
>>>>>> + - System recipes and classes: they are responsible for setting
>>>>>> up Debian-like infrastructure, manages Debian tools execution for
>>>>>> package building and installation, generation root file system
>>>>>> images
>>>>>> + - User recipes: custom user applications, that should be builе
>>>>>> from sources +
>>>>>> +There are two types of dependencies in Isar:
>>>>>> + - Dependency between bitbake recipes
>>>>>> + - Dependencies in Debian filesystem
>>>>>> +
>>>>>> +**NOTE:** Isar doesn't manage dependencies in Debian file
>>>>>> systems. User can only list specific Debian dependencies in
>>>>>> recipe, so they eventually will be passed to apt-get or
>>>>>> multistrap. Dependency installation is managed by Debian tools.
>>>>>> + +### 2.2 Stamps + +Each task managed by bitbake uses stamp to
>>>>>> notify that it has been completed. Due to Isar supports various
>>>>>> Debian distributions and parallel builds for multiple machines
>>>>>> and architectures, the stamps for tasks use special suffixes
>>>>>> that include:
>>>>>> + - Debian distro name
>>>>>> + - Architecture
>>>>>> + - Machine
>>>>>> +
>>>>>> +Typical example, when Isar builds the following configurations:
>>>>>> + - Debian Jessie, amd64
>>>>>> + - Debian Jessie, i386
>>>>>> + - Debian Stretch, i386
>>>>>> +
>>>>>> +In this case there will be 3 different buildchroots, so standard
>>>>>> hello demo application should be processed 3 times for each
>>>>>> environment. Three different sets of stamps should be used for
>>>>>> correct bitbake operating. + +### 2.3 Buildchroot +
>>>>>> +One of the key aspect of Debian philosophy claims the fact, that
>>>>>> everything in Debian should be built within Debian environment.
>>>>>> Moreover native compilation is more preferable than
>>>>>> cross-compilation. To follow this rules, Isar introduces the new
>>>>>> component - buildchroot. Bulidchroot is typical Debian filesystem
>>>>>> that is created using standard Debian tools: multistrap, apt. The
>>>>>> source of packages can be either official Debian repositories or
>>>>>> custom repositories created by user. + +Buildchroot lifecycle can
>>>>>> be described as following:
>>>>>> + - Buildchroot has initial configuration file which is passed to
>>>>>> multistrap tool. This configuration file is generated by bitbake
>>>>>> recipe from patterns and values defined by user. Based on this
>>>>>> configuration file, multistrap generates initial filesystem.
>>>>>> + - During building custom Debian package, list of its build
>>>>>> dependencies is installed to buildchroot.
>>>>>> + - When package has been built, it's installed to current
>>>>>> buildchroot to satisfy further packages build dependencies. +
>>>>>> +### 2.4 Target Root Filesystem
>>>>>> +
>>>>>> +Target filesystem is quite similar to buildchroot. The only
>>>>>> difference is that it doesn't have development packages
>>>>>> installed.
>>>>>> + +Target filesystem lifecycle can be described as following:
>>>>>> + - Target filesystem has initial configuration file which is
>>>>>> passed to multistrap tool. This configuration file is generated
>>>>>> by bitbake recipe from patterns and values defined by user.
>>>>>> Based on this configuration file, multistrap generates initial
>>>>>> filesystem.
>>>>>> + - According to the list of custom packages in bitbake recipes,
>>>>>> the initial filesystem will be populated by successfully built
>>>>>> packages. + +# 3 Isar Internal Processes
>>>>>> +
>>>>>> +### 3.1 General Overview
>>>>>> +
>>>>>> +Whole Isar build process can be split into the following steps:
>>>>>> + - Generation of initial buildchroots for each configuration
>>>>>> (Debian distro, machine and architecture) requested by user.
>>>>>> + - Generation of initial target filesystems for each
>>>>>> configuration.
>>>>>> + - Building custom packages.
>>>>>> + - Populate target filesystems.
>>>>>> + - Generate bootable images.
>>>>>> +
>>>>>> +All these steps are described in details below.
>>>>>> +
>>>>>> +### 3.2 Initial Buildchroot Generation
>>>>>> +
>>>>>> +As mentioned above, initial buildchroot is generated using
>>>>>> multistrap. The bitbake recipe which is responsible for
>>>>>> buildchroot can be found here:
>>>>>> `meta/recipes-devtools/buildchroot/buildchroot.bb`
>>>>>> + +This recipe implementes `do_build` task which performs the
>>>>>> following: +1. Generates multistrap config from template:
>>>>>> `meta/recipes-devtools/buildchroot/files/multistrap.conf.in` +2.
>>>>>> Install pre/post scripts for multistrap:
>>>>>> `meta/recipes-devtools/buildchroot/files/configscript.sh` and
>>>>>> `meta/recipes-devtools/buildchroot/files/setup.sh` +3. Run
>>>>>> multistrap +4. Install script for building custom Debian
>>>>>> packages: `meta/recipes-devtools/buildchroot/files/build.sh` +
>>>>>> +The single stamp is created for each user buildchroot
>>>>>> configuration. + +### 3.3 Initial Target Filesystem Generation +
>>>>>> +Initial target filesystem generation process is very similar to
>>>>>> buildchroot creating, the difference is only in initial packages
>>>>>> list. + +Target image recipes are the part of Isar core. There
>>>>>> is a sample of typical Isar image that can be customized
>>>>>> according to the user requirements:
>>>>>> `meta-isar/recipes-core/images/isar-image-base.bb` +Like for
>>>>>> buildchroot, the multistrap configuration files for image can be
>>>>>> found here: `meta-isar/recipes-core/images/files`, and it
>>>>>> implements `do_build` task. + +### 3.4 Building Custom Packages
>>>>>> + +Isar provides possibility to build Debian packages from
>>>>>> sources. This features works with Debian-like source packages,
>>>>>> i.e. the source code tree should contain debian folder. The
>>>>>> build process is implemented in `meta/classes/dpkg.bbclass` and
>>>>>> consists from the following steps: +1. Task `do_fetch`: fetch
>>>>>> source code from external link +2. Task `do_unpack`: unpack
>>>>>> source code to `${BUILDCHROOT_DIR}/home/build/${PN}` +3. Task
>>>>>> `do_build`: switch to buildchroot using chroot command and run
>>>>>> `build.sh` script. The `build.sh` script performs the following:
>>>>>> +   1. Go to `/home/build/${PN}`
>>>>>> +   2. Get list of dependencies from debian/control and install
>>>>>> them using apt.
>>>>>> +   3. Run dpkg-buildpackage
>>>>>> +4. Task `do_install`: install successfully built packages
>>>>>> `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy directory
>>>>>> `${DEPLOY_DIR_DEB}` + +### 3.5 Populate Target Filesystem
>>>>>> +
>>>>>> +Each target image can be extended by custom packages listed in
>>>>>> IMAGE_INSTALL variable. Task `do_populate` performs the
>>>>>> following: +1. Parse IMAGE_INSTALL variable +2. Find respective
>>>>>> packages in `${DEPLOY_DIR_DEB}` +3. Copy them to deb folder in
>>>>>> dedicated target filesystem +4. Execute dpkg command in chroot
>>>>>> for all the copied packages +
>>>>>> +### 3.6 Generate Bootable Image
>>>>>> +
>>>>>> +This process contains the following steps:
>>>>>> +1. Task `do_ext4_image`: target filesystem is packed to extfs
>>>>>> image +2. wic tool generates bootable image for dedicated
>>>>>> platform
>>>>>       
>>>    
>>
> 

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov