Re: [PATCH] doc: Add technical overview

[Henning Schild <henning.schild@siemens.com>]

Am Tue, 8 Aug 2017 16:12:02 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> On 08/08/2017 04:03 PM, Henning Schild wrote:
> > Am Tue, 8 Aug 2017 15:41:10 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >   
> >> On 08/08/2017 02:54 PM, Henning Schild wrote:  
> >>> Am Tue, 8 Aug 2017 14:32:00 +0300
> >>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >>>      
> >>>> Hi,
> >>>>
> >>>> On 08/08/2017 02:11 PM, Henning Schild wrote:  
> >>>>> I did not really read this yet. At the moment i think we have
> >>>>> too many changes in the q to add such a document to the
> >>>>> repository. Any change would just mean heaving to keep it up to
> >>>>> date. We talked about the need for such a document for the
> >>>>> possible OE integration. I think it is valuable for that
> >>>>> already. 
> >>>>
> >>>> Briefly speaking, this document describes initial motivation and
> >>>> high-level current overview.
> >>>>     
> >>>>> That should be added after we all are happy with the feature-set
> >>>>> of Isar, until then it would just be another file to update when
> >>>>> changing how things work.  
> >>>>
> >>>> Hmm, this document doesn't affect any discussions about Isar
> >>>> feature set. It's intended to be as technical background for
> >>>> hot-plug joining to this project. During discussions last month I
> >>>> see that we have gap in our understandings how Isar works, so
> >>>> this document should be start point to eliminate such gaps.
> >>>>
> >>>> For sure, it's *not* a specification, which explicitly defines
> >>>> specific feature set and nothing more will be added. It's just an
> >>>> explanation of current state. I see no problems to regularly
> >>>> update it, moreover I highly appreciate this. Lack of
> >>>> documentation is the most common problem of open source
> >>>> projects, so as soon we start documenting the things, the easier
> >>>> will be documentation maintenance in future.  
> >>>
> >>> For sure we need that! But especially section 3 contains lots of
> >>> interna that will change if my patches get accepted. That is why i
> >>> would like to postpone adding the file after my patches went in.  
> >>
> >> As I said, this document describes current project state, so I
> >> don't think that upcoming changes should block publishing of
> >> documentation.
> >>
> >>  
> >>>
> >>> Adding that file before them would mean another rebase round for
> >>> documentation. I would be happy to rebase this one for you on top
> >>> of a master that contains my stuff, but not too happy about the
> >>> other way around.  
> >>
> >> The patch creates document from the scratch, so it can't conflict
> >> with any upcoming patches.  
> > 
> > It conflicts with other patches that are under review right now.
> > All i am asking is to merge conflicting patches in the order the
> > patches initially showed up for review, in that case this one comes
> > last anyways. And when its time has come it will be outdated, and i
> > will update it for you.
> >   
> 
> Hmm, are you speaking about exactly this patch or about previous
> series?

Section 3.4 and 3.5 conflict with my changes, they will have to be
changed. If your patch comes first i would have to rebase.

i.e.
- 3.4 talks about dpkg.bbclass which i renamed
[PATCH 11-16 of 16 v2 4/6]
- 3.5 will not happen anymore if everything enters the rootfs through
  multistrap
[PATCH 3/3] classes: image: remove populate and replace it with a
custom repo


Henning


> Alex
> 
> >> Also I expect, that the author of new features will update the
> >> documentation, so some reference document should be published
> >> anyway.  
> > 
> > Sure.
> >   
> >> To be honest I'm not happy with the idea that I'll document all the
> >> new features in future.  
> > 
> > Missing or inconsistent documentation are a valid reason to not
> > accept a patch so it will not come to that.
> > 
> > Henning
> >   
> >> Alex
> >>  
> >>>>>
> >>>>> Am Tue, 8 Aug 2017 13:05:59 +0300
> >>>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >>>>>         
> >>>>>> Add initial Isar technical overview.
> >>>>>>
> >>>>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> >>>>>> ---
> >>>>>>     doc/technical_overview.md | 110
> >>>>>> ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed,
> >>>>>> 110 insertions(+) create mode 100644 doc/technical_overview.md
> >>>>>>
> >>>>>> diff --git a/doc/technical_overview.md
> >>>>>> b/doc/technical_overview.md new file mode 100644
> >>>>>> index 0000000..f65fc08
> >>>>>> --- /dev/null
> >>>>>> +++ b/doc/technical_overview.md
> >>>>>> @@ -0,0 +1,110 @@
> >>>>>> +# 1 Introduction
> >>>>>> +
> >>>>>> +Isar can be shortly described as a set of bitbake recipes that
> >>>>>> implement main build system logic. To simplify overall Isar
> >>>>>> understanding, this document is split into two main parts:
> >>>>>> + - Isar logical components
> >>>>>> + - Isar internal processes
> >>>>>> +
> >>>>>> +# 2 Isar Logical Components
> >>>>>> +
> >>>>>> +In this chapter the most important Isar components are
> >>>>>> considered in details. In this text component doesn't
> >>>>>> especially mean some self-contained single entity of
> >>>>>> something, it's just an attempt to split Isar internals by
> >>>>>> various criteria. + +### 2.1 Bitbake and Recipes + +All the
> >>>>>> processes in Isar are started by bitbake, so it manages
> >>>>>> general build executing process. Recipes in Isar can be split
> >>>>>> in two categories: +
> >>>>>> + - System recipes and classes: they are responsible for
> >>>>>> setting up Debian-like infrastructure, manages Debian tools
> >>>>>> execution for package building and installation, generation
> >>>>>> root file system images
> >>>>>> + - User recipes: custom user applications, that should be
> >>>>>> builе from sources +
> >>>>>> +There are two types of dependencies in Isar:
> >>>>>> + - Dependency between bitbake recipes
> >>>>>> + - Dependencies in Debian filesystem
> >>>>>> +
> >>>>>> +**NOTE:** Isar doesn't manage dependencies in Debian file
> >>>>>> systems. User can only list specific Debian dependencies in
> >>>>>> recipe, so they eventually will be passed to apt-get or
> >>>>>> multistrap. Dependency installation is managed by Debian tools.
> >>>>>> + +### 2.2 Stamps + +Each task managed by bitbake uses stamp to
> >>>>>> notify that it has been completed. Due to Isar supports various
> >>>>>> Debian distributions and parallel builds for multiple machines
> >>>>>> and architectures, the stamps for tasks use special suffixes
> >>>>>> that include:
> >>>>>> + - Debian distro name
> >>>>>> + - Architecture
> >>>>>> + - Machine
> >>>>>> +
> >>>>>> +Typical example, when Isar builds the following
> >>>>>> configurations:
> >>>>>> + - Debian Jessie, amd64
> >>>>>> + - Debian Jessie, i386
> >>>>>> + - Debian Stretch, i386
> >>>>>> +
> >>>>>> +In this case there will be 3 different buildchroots, so
> >>>>>> standard hello demo application should be processed 3 times
> >>>>>> for each environment. Three different sets of stamps should be
> >>>>>> used for correct bitbake operating. + +### 2.3 Buildchroot +
> >>>>>> +One of the key aspect of Debian philosophy claims the fact,
> >>>>>> that everything in Debian should be built within Debian
> >>>>>> environment. Moreover native compilation is more preferable
> >>>>>> than cross-compilation. To follow this rules, Isar introduces
> >>>>>> the new component - buildchroot. Bulidchroot is typical Debian
> >>>>>> filesystem that is created using standard Debian tools:
> >>>>>> multistrap, apt. The source of packages can be either official
> >>>>>> Debian repositories or custom repositories created by user. +
> >>>>>> +Buildchroot lifecycle can be described as following:
> >>>>>> + - Buildchroot has initial configuration file which is passed
> >>>>>> to multistrap tool. This configuration file is generated by
> >>>>>> bitbake recipe from patterns and values defined by user. Based
> >>>>>> on this configuration file, multistrap generates initial
> >>>>>> filesystem.
> >>>>>> + - During building custom Debian package, list of its build
> >>>>>> dependencies is installed to buildchroot.
> >>>>>> + - When package has been built, it's installed to current
> >>>>>> buildchroot to satisfy further packages build dependencies. +
> >>>>>> +### 2.4 Target Root Filesystem
> >>>>>> +
> >>>>>> +Target filesystem is quite similar to buildchroot. The only
> >>>>>> difference is that it doesn't have development packages
> >>>>>> installed.
> >>>>>> + +Target filesystem lifecycle can be described as following:
> >>>>>> + - Target filesystem has initial configuration file which is
> >>>>>> passed to multistrap tool. This configuration file is generated
> >>>>>> by bitbake recipe from patterns and values defined by user.
> >>>>>> Based on this configuration file, multistrap generates initial
> >>>>>> filesystem.
> >>>>>> + - According to the list of custom packages in bitbake
> >>>>>> recipes, the initial filesystem will be populated by
> >>>>>> successfully built packages. + +# 3 Isar Internal Processes
> >>>>>> +
> >>>>>> +### 3.1 General Overview
> >>>>>> +
> >>>>>> +Whole Isar build process can be split into the following
> >>>>>> steps:
> >>>>>> + - Generation of initial buildchroots for each configuration
> >>>>>> (Debian distro, machine and architecture) requested by user.
> >>>>>> + - Generation of initial target filesystems for each
> >>>>>> configuration.
> >>>>>> + - Building custom packages.
> >>>>>> + - Populate target filesystems.
> >>>>>> + - Generate bootable images.
> >>>>>> +
> >>>>>> +All these steps are described in details below.
> >>>>>> +
> >>>>>> +### 3.2 Initial Buildchroot Generation
> >>>>>> +
> >>>>>> +As mentioned above, initial buildchroot is generated using
> >>>>>> multistrap. The bitbake recipe which is responsible for
> >>>>>> buildchroot can be found here:
> >>>>>> `meta/recipes-devtools/buildchroot/buildchroot.bb`
> >>>>>> + +This recipe implementes `do_build` task which performs the
> >>>>>> following: +1. Generates multistrap config from template:
> >>>>>> `meta/recipes-devtools/buildchroot/files/multistrap.conf.in`
> >>>>>> +2. Install pre/post scripts for multistrap:
> >>>>>> `meta/recipes-devtools/buildchroot/files/configscript.sh` and
> >>>>>> `meta/recipes-devtools/buildchroot/files/setup.sh` +3. Run
> >>>>>> multistrap +4. Install script for building custom Debian
> >>>>>> packages: `meta/recipes-devtools/buildchroot/files/build.sh` +
> >>>>>> +The single stamp is created for each user buildchroot
> >>>>>> configuration. + +### 3.3 Initial Target Filesystem Generation
> >>>>>> + +Initial target filesystem generation process is very
> >>>>>> similar to buildchroot creating, the difference is only in
> >>>>>> initial packages list. + +Target image recipes are the part of
> >>>>>> Isar core. There is a sample of typical Isar image that can be
> >>>>>> customized according to the user requirements:
> >>>>>> `meta-isar/recipes-core/images/isar-image-base.bb` +Like for
> >>>>>> buildchroot, the multistrap configuration files for image can
> >>>>>> be found here: `meta-isar/recipes-core/images/files`, and it
> >>>>>> implements `do_build` task. + +### 3.4 Building Custom Packages
> >>>>>> + +Isar provides possibility to build Debian packages from
> >>>>>> sources. This features works with Debian-like source packages,
> >>>>>> i.e. the source code tree should contain debian folder. The
> >>>>>> build process is implemented in `meta/classes/dpkg.bbclass` and
> >>>>>> consists from the following steps: +1. Task `do_fetch`: fetch
> >>>>>> source code from external link +2. Task `do_unpack`: unpack
> >>>>>> source code to `${BUILDCHROOT_DIR}/home/build/${PN}` +3. Task
> >>>>>> `do_build`: switch to buildchroot using chroot command and run
> >>>>>> `build.sh` script. The `build.sh` script performs the
> >>>>>> following:
> >>>>>> +   1. Go to `/home/build/${PN}`
> >>>>>> +   2. Get list of dependencies from debian/control and install
> >>>>>> them using apt.
> >>>>>> +   3. Run dpkg-buildpackage
> >>>>>> +4. Task `do_install`: install successfully built packages
> >>>>>> `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy directory
> >>>>>> `${DEPLOY_DIR_DEB}` + +### 3.5 Populate Target Filesystem
> >>>>>> +
> >>>>>> +Each target image can be extended by custom packages listed in
> >>>>>> IMAGE_INSTALL variable. Task `do_populate` performs the
> >>>>>> following: +1. Parse IMAGE_INSTALL variable +2. Find respective
> >>>>>> packages in `${DEPLOY_DIR_DEB}` +3. Copy them to deb folder in
> >>>>>> dedicated target filesystem +4. Execute dpkg command in chroot
> >>>>>> for all the copied packages +
> >>>>>> +### 3.6 Generate Bootable Image
> >>>>>> +
> >>>>>> +This process contains the following steps:
> >>>>>> +1. Task `do_ext4_image`: target filesystem is packed to extfs
> >>>>>> image +2. wic tool generates bootable image for dedicated
> >>>>>> platform  
> >>>>>         
> >>>      
> >>  
> >   
>