From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6451845424769662976 X-Received: by 10.107.22.194 with SMTP id 185mr3150271iow.13.1502218748928; Tue, 08 Aug 2017 11:59:08 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.36.29.194 with SMTP id 185ls7515048itj.17.gmail; Tue, 08 Aug 2017 11:59:08 -0700 (PDT) X-Received: by 10.129.131.150 with SMTP id t144mr3391288ywf.15.1502218748558; Tue, 08 Aug 2017 11:59:08 -0700 (PDT) Received: by 10.55.101.138 with SMTP id z132msqkb; Tue, 8 Aug 2017 04:32:08 -0700 (PDT) X-Received: by 10.28.99.135 with SMTP id x129mr553594wmb.9.1502191928498; Tue, 08 Aug 2017 04:32:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502191928; cv=none; d=google.com; s=arc-20160816; b=bG8HzeEXq+C9EzEErquJv9g9QggVjQYxD7r9AHjXkHTIzgWEs1RYibq4JSEaWVd/Ba u4u7Zafc58FOK/Xj2teVm+deb4pO033NxTS+ub47+yz91tItJNeiP7jTYrl5av3AGoXn vQ+3SN/tnjrTEUDHJ/3e/0aLG0N8G0B/kZZsAPGGdQXBkAqZMehCj7NmZAvbkPRA1fqp /RHECd5/WHb+IM1ZfxHgTWLbkbXp4TRLasBeH2ujT3/7lZkCnwgTssEmlK2WNoEWwP6y OxtS/Abx/eKuGSV4i405g1H5dtk4cCUQ8bluPNz8tT4uFIVCUGZHh6D7bVkuQUzFezd0 FMJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject :arc-authentication-results; bh=iwldOazxHSNdYro3tQkJd+T5y3rSzjSqIWIYvcu7+5k=; b=Dx2AzBDr2c7Mu/i4h5diui5pHbfpVXAGzHYiOo7qUKp86Lt4LGJSewFVMyK7u3Ql0u yBRnTWD2Ml7DyzOw9/A5hyGA0g7dVXGm6Sin2h3FDU3MBXGwlLGYEeSFU4ySZ7g/1IgH VSyqV3vKDJEZ5DrPmn4RmSjms6qzWtZMqNhmq7ybv8XMwPnafPUGd1XJ4i3LuLbEFZ5U jUJBYkxtFKt9+LyPG6WP8g1ZP13ZV8iv1Mh3L53WuV4OSUJZ/FSmTVwHRPow1NdGN5/W tqWpZFREoY4xeRWk31goTxpH2K/zNALbxQPBTJ4XWIpD5VuBaJNHnTO9QYgbArOih3PD EVxw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id r65si411783wmf.2.2017.08.08.04.32.08 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 08 Aug 2017 04:32:08 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Received: from [10.0.2.15] ([188.227.110.165]) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id v78BW5fG022166 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Tue, 8 Aug 2017 13:32:07 +0200 Subject: Re: [PATCH] doc: Add technical overview To: Henning Schild Cc: isar-users@googlegroups.com References: <20170808100559.19682-1-asmirnov@ilbers.de> <20170808131102.0ef91da0@md1em3qc> From: Alexander Smirnov Message-ID: Date: Tue, 8 Aug 2017 14:32:00 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: <20170808131102.0ef91da0@md1em3qc> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: 6mbWvqPFrkGL Hi, On 08/08/2017 02:11 PM, Henning Schild wrote: > I did not really read this yet. At the moment i think we have too many > changes in the q to add such a document to the repository. Any change > would just mean heaving to keep it up to date. > We talked about the need for such a document for the possible OE > integration. I think it is valuable for that already. > Briefly speaking, this document describes initial motivation and high-level current overview. > That should be added after we all are happy with the feature-set of > Isar, until then it would just be another file to update when changing > how things work. Hmm, this document doesn't affect any discussions about Isar feature set. It's intended to be as technical background for hot-plug joining to this project. During discussions last month I see that we have gap in our understandings how Isar works, so this document should be start point to eliminate such gaps. For sure, it's *not* a specification, which explicitly defines specific feature set and nothing more will be added. It's just an explanation of current state. I see no problems to regularly update it, moreover I highly appreciate this. Lack of documentation is the most common problem of open source projects, so as soon we start documenting the things, the easier will be documentation maintenance in future. Alex > > Am Tue, 8 Aug 2017 13:05:59 +0300 > schrieb Alexander Smirnov : > >> Add initial Isar technical overview. >> >> Signed-off-by: Alexander Smirnov >> --- >> doc/technical_overview.md | 110 >> ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 110 >> insertions(+) create mode 100644 doc/technical_overview.md >> >> diff --git a/doc/technical_overview.md b/doc/technical_overview.md >> new file mode 100644 >> index 0000000..f65fc08 >> --- /dev/null >> +++ b/doc/technical_overview.md >> @@ -0,0 +1,110 @@ >> +# 1 Introduction >> + >> +Isar can be shortly described as a set of bitbake recipes that >> implement main build system logic. To simplify overall Isar >> understanding, this document is split into two main parts: >> + - Isar logical components >> + - Isar internal processes >> + >> +# 2 Isar Logical Components >> + >> +In this chapter the most important Isar components are considered in >> details. In this text component doesn't especially mean some >> self-contained single entity of something, it's just an attempt to >> split Isar internals by various criteria. + +### 2.1 Bitbake and >> Recipes + +All the processes in Isar are started by bitbake, so it >> manages general build executing process. Recipes in Isar can be split >> in two categories: + >> + - System recipes and classes: they are responsible for setting up >> Debian-like infrastructure, manages Debian tools execution for >> package building and installation, generation root file system images >> + - User recipes: custom user applications, that should be builŠµ from >> sources + >> +There are two types of dependencies in Isar: >> + - Dependency between bitbake recipes >> + - Dependencies in Debian filesystem >> + >> +**NOTE:** Isar doesn't manage dependencies in Debian file systems. >> User can only list specific Debian dependencies in recipe, so they >> eventually will be passed to apt-get or multistrap. Dependency >> installation is managed by Debian tools. + +### 2.2 Stamps + >> +Each task managed by bitbake uses stamp to notify that it has been >> completed. Due to Isar supports various Debian distributions and >> parallel builds for multiple machines and architectures, the stamps >> for tasks use special suffixes that include: >> + - Debian distro name >> + - Architecture >> + - Machine >> + >> +Typical example, when Isar builds the following configurations: >> + - Debian Jessie, amd64 >> + - Debian Jessie, i386 >> + - Debian Stretch, i386 >> + >> +In this case there will be 3 different buildchroots, so standard >> hello demo application should be processed 3 times for each >> environment. Three different sets of stamps should be used for >> correct bitbake operating. + +### 2.3 Buildchroot + >> +One of the key aspect of Debian philosophy claims the fact, that >> everything in Debian should be built within Debian environment. >> Moreover native compilation is more preferable than >> cross-compilation. To follow this rules, Isar introduces the new >> component - buildchroot. Bulidchroot is typical Debian filesystem >> that is created using standard Debian tools: multistrap, apt. The >> source of packages can be either official Debian repositories or >> custom repositories created by user. + +Buildchroot lifecycle can be >> described as following: >> + - Buildchroot has initial configuration file which is passed to >> multistrap tool. This configuration file is generated by bitbake >> recipe from patterns and values defined by user. Based on this >> configuration file, multistrap generates initial filesystem. >> + - During building custom Debian package, list of its build >> dependencies is installed to buildchroot. >> + - When package has been built, it's installed to current >> buildchroot to satisfy further packages build dependencies. + >> +### 2.4 Target Root Filesystem >> + >> +Target filesystem is quite similar to buildchroot. The only >> difference is that it doesn't have development packages installed. + >> +Target filesystem lifecycle can be described as following: >> + - Target filesystem has initial configuration file which is passed >> to multistrap tool. This configuration file is generated by bitbake >> recipe from patterns and values defined by user. Based on this >> configuration file, multistrap generates initial filesystem. >> + - According to the list of custom packages in bitbake recipes, the >> initial filesystem will be populated by successfully built packages. + >> +# 3 Isar Internal Processes >> + >> +### 3.1 General Overview >> + >> +Whole Isar build process can be split into the following steps: >> + - Generation of initial buildchroots for each configuration (Debian >> distro, machine and architecture) requested by user. >> + - Generation of initial target filesystems for each configuration. >> + - Building custom packages. >> + - Populate target filesystems. >> + - Generate bootable images. >> + >> +All these steps are described in details below. >> + >> +### 3.2 Initial Buildchroot Generation >> + >> +As mentioned above, initial buildchroot is generated using >> multistrap. The bitbake recipe which is responsible for buildchroot >> can be found here: `meta/recipes-devtools/buildchroot/buildchroot.bb` >> + +This recipe implementes `do_build` task which performs the >> following: +1. Generates multistrap config from template: >> `meta/recipes-devtools/buildchroot/files/multistrap.conf.in` +2. >> Install pre/post scripts for multistrap: >> `meta/recipes-devtools/buildchroot/files/configscript.sh` and >> `meta/recipes-devtools/buildchroot/files/setup.sh` +3. Run multistrap >> +4. Install script for building custom Debian packages: >> `meta/recipes-devtools/buildchroot/files/build.sh` + +The single >> stamp is created for each user buildchroot configuration. + +### 3.3 >> Initial Target Filesystem Generation + +Initial target filesystem >> generation process is very similar to buildchroot creating, the >> difference is only in initial packages list. + +Target image recipes >> are the part of Isar core. There is a sample of typical Isar image >> that can be customized according to the user requirements: >> `meta-isar/recipes-core/images/isar-image-base.bb` +Like for >> buildchroot, the multistrap configuration files for image can be >> found here: `meta-isar/recipes-core/images/files`, and it implements >> `do_build` task. + +### 3.4 Building Custom Packages + +Isar provides >> possibility to build Debian packages from sources. This features >> works with Debian-like source packages, i.e. the source code tree >> should contain debian folder. The build process is implemented in >> `meta/classes/dpkg.bbclass` and consists from the following steps: >> +1. Task `do_fetch`: fetch source code from external link +2. Task >> `do_unpack`: unpack source code to >> `${BUILDCHROOT_DIR}/home/build/${PN}` +3. Task `do_build`: switch to >> buildchroot using chroot command and run `build.sh` script. The >> `build.sh` script performs the following: >> + 1. Go to `/home/build/${PN}` >> + 2. Get list of dependencies from debian/control and install them >> using apt. >> + 3. Run dpkg-buildpackage >> +4. Task `do_install`: install successfully built packages >> `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy directory >> `${DEPLOY_DIR_DEB}` + +### 3.5 Populate Target Filesystem >> + >> +Each target image can be extended by custom packages listed in >> IMAGE_INSTALL variable. Task `do_populate` performs the following: >> +1. Parse IMAGE_INSTALL variable +2. Find respective packages in >> `${DEPLOY_DIR_DEB}` +3. Copy them to deb folder in dedicated target >> filesystem +4. Execute dpkg command in chroot for all the copied >> packages + >> +### 3.6 Generate Bootable Image >> + >> +This process contains the following steps: >> +1. Task `do_ext4_image`: target filesystem is packed to extfs image >> +2. wic tool generates bootable image for dedicated platform >