From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6451845424769662976 X-Received: by 10.223.175.195 with SMTP id y3mr352738wrd.29.1502190545883; Tue, 08 Aug 2017 04:09:05 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.127.73 with SMTP id a70ls902114wmd.26.canary-gmail; Tue, 08 Aug 2017 04:09:05 -0700 (PDT) X-Received: by 10.28.125.142 with SMTP id y136mr543454wmc.3.1502190545561; Tue, 08 Aug 2017 04:09:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502190545; cv=none; d=google.com; s=arc-20160816; b=FWCrAd8CCW02AwQPfeiG+l+urV3riIkmNj1roV+phLILZttDN4NC7Q3K9WhaQEoSQe aoIev0CHk/76EqC2xREx6CSyKvEArJn+iuvH/cWt8K50MZBYgMdFlr94eXnmgtdeO3oG 8amcBshv6GG2fzER2QlnGx54oQzQKF+Vju3taAcla9x1B1ZVlEvNRz8jUSKePpzfNtwR Tadveo8QwL6ebjyLWHj+uGIkInDAZmTLYCNzdrGW84y1vZxPWuCqX2qVcRmoAlK+CYR8 tKWQjo8b8kiBzGvE5cYokmvwy+PlQPZkz31IH/29h8vmDi6t8LHvYQDh40MT2JzfNMLy 4UWQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:arc-authentication-results; bh=VzVLYtkl5GEpzmY2pNxLw0UOVJ9JGqmZwAAzALbD8Jw=; b=wCqBaYzEs+fkZFjlMd/wc42haO/zFQ57NSiKRdj9rRLE0VCpZToo8XGyx4m9Q8LnSZ dWTfMF+Td5dwYWboGHYMZrkmDka5V/oi7aEseMZyWDTybBu98a3PEj6JJDn9Vy+MLxro Zyp0UJNuwECQDMLZ2nhl9CjFRcmW7wLlrBFtWMzKApjJGcLxEFtKRQ7G8SL3L1QUODVV zqrzGVDOJMzgW0jdVxU5pKCbbrc9nPER4l7XH3UGeuMwjicH0GJxTLuOEcHTHspalBRO 5g7g+SgZzOk0Mvxc/jE8BHI2T+yfM30otrj3ziz+TvPkjcLI7z3VWfbz4slC2nYFShTX DD2Q== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 192.35.17.28 is neither permitted nor denied by best guess record for domain of henning.schild@siemens.com) smtp.mailfrom=henning.schild@siemens.com Return-Path: Received: from goliath.siemens.de (goliath.siemens.de. [192.35.17.28]) by gmr-mx.google.com with ESMTPS id x7si354077wmf.8.2017.08.08.04.09.05 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 08 Aug 2017 04:09:05 -0700 (PDT) Received-SPF: neutral (google.com: 192.35.17.28 is neither permitted nor denied by best guess record for domain of henning.schild@siemens.com) client-ip=192.35.17.28; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 192.35.17.28 is neither permitted nor denied by best guess record for domain of henning.schild@siemens.com) smtp.mailfrom=henning.schild@siemens.com Received: from mail1.siemens.de (mail1.siemens.de [139.23.33.14]) by goliath.siemens.de (8.15.2/8.15.2) with ESMTPS id v78B95Ue010334 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 8 Aug 2017 13:09:05 +0200 Received: from md1em3qc ([139.25.68.40]) by mail1.siemens.de (8.15.2/8.15.2) with ESMTP id v78B95B5021614; Tue, 8 Aug 2017 13:09:05 +0200 Date: Tue, 8 Aug 2017 13:11:02 +0200 From: Henning Schild To: Alexander Smirnov Cc: Subject: Re: [PATCH] doc: Add technical overview Message-ID: <20170808131102.0ef91da0@md1em3qc> In-Reply-To: <20170808100559.19682-1-asmirnov@ilbers.de> References: <20170808100559.19682-1-asmirnov@ilbers.de> X-Mailer: Claws Mail 3.13.2 (GTK+ 2.24.31; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-TUID: p6PUl2auax0r I did not really read this yet. At the moment i think we have too many changes in the q to add such a document to the repository. Any change would just mean heaving to keep it up to date. We talked about the need for such a document for the possible OE integration. I think it is valuable for that already. That should be added after we all are happy with the feature-set of Isar, until then it would just be another file to update when changing how things work. Henning Am Tue, 8 Aug 2017 13:05:59 +0300 schrieb Alexander Smirnov : > Add initial Isar technical overview. >=20 > Signed-off-by: Alexander Smirnov > --- > doc/technical_overview.md | 110 > ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 110 > insertions(+) create mode 100644 doc/technical_overview.md >=20 > diff --git a/doc/technical_overview.md b/doc/technical_overview.md > new file mode 100644 > index 0000000..f65fc08 > --- /dev/null > +++ b/doc/technical_overview.md > @@ -0,0 +1,110 @@ > +# 1 Introduction > + > +Isar can be shortly described as a set of bitbake recipes that > implement main build system logic. To simplify overall Isar > understanding, this document is split into two main parts: > + - Isar logical components > + - Isar internal processes > + > +# 2 Isar Logical Components > + > +In this chapter the most important Isar components are considered in > details. In this text component doesn't especially mean some > self-contained single entity of something, it's just an attempt to > split Isar internals by various criteria. + +### 2.1 Bitbake and > Recipes + +All the processes in Isar are started by bitbake, so it > manages general build executing process. Recipes in Isar can be split > in two categories: + > + - System recipes and classes: they are responsible for setting up > Debian-like infrastructure, manages Debian tools execution for > package building and installation, generation root file system images > + - User recipes: custom user applications, that should be buil=D0=B5 from > sources + > +There are two types of dependencies in Isar: > + - Dependency between bitbake recipes > + - Dependencies in Debian filesystem > + > +**NOTE:** Isar doesn't manage dependencies in Debian file systems. > User can only list specific Debian dependencies in recipe, so they > eventually will be passed to apt-get or multistrap. Dependency > installation is managed by Debian tools. + +### 2.2 Stamps + > +Each task managed by bitbake uses stamp to notify that it has been > completed. Due to Isar supports various Debian distributions and > parallel builds for multiple machines and architectures, the stamps > for tasks use special suffixes that include: > + - Debian distro name > + - Architecture > + - Machine > + > +Typical example, when Isar builds the following configurations: > + - Debian Jessie, amd64 > + - Debian Jessie, i386 > + - Debian Stretch, i386 > + > +In this case there will be 3 different buildchroots, so standard > hello demo application should be processed 3 times for each > environment. Three different sets of stamps should be used for > correct bitbake operating. + +### 2.3 Buildchroot + > +One of the key aspect of Debian philosophy claims the fact, that > everything in Debian should be built within Debian environment. > Moreover native compilation is more preferable than > cross-compilation. To follow this rules, Isar introduces the new > component - buildchroot. Bulidchroot is typical Debian filesystem > that is created using standard Debian tools: multistrap, apt. The > source of packages can be either official Debian repositories or > custom repositories created by user. + +Buildchroot lifecycle can be > described as following: > + - Buildchroot has initial configuration file which is passed to > multistrap tool. This configuration file is generated by bitbake > recipe from patterns and values defined by user. Based on this > configuration file, multistrap generates initial filesystem. > + - During building custom Debian package, list of its build > dependencies is installed to buildchroot. > + - When package has been built, it's installed to current > buildchroot to satisfy further packages build dependencies. + > +### 2.4 Target Root Filesystem > + > +Target filesystem is quite similar to buildchroot. The only > difference is that it doesn't have development packages installed. + > +Target filesystem lifecycle can be described as following: > + - Target filesystem has initial configuration file which is passed > to multistrap tool. This configuration file is generated by bitbake > recipe from patterns and values defined by user. Based on this > configuration file, multistrap generates initial filesystem. > + - According to the list of custom packages in bitbake recipes, the > initial filesystem will be populated by successfully built packages. + > +# 3 Isar Internal Processes > + > +### 3.1 General Overview > + > +Whole Isar build process can be split into the following steps: > + - Generation of initial buildchroots for each configuration (Debian > distro, machine and architecture) requested by user. > + - Generation of initial target filesystems for each configuration. > + - Building custom packages. > + - Populate target filesystems. > + - Generate bootable images. > + > +All these steps are described in details below. > + > +### 3.2 Initial Buildchroot Generation > + > +As mentioned above, initial buildchroot is generated using > multistrap. The bitbake recipe which is responsible for buildchroot > can be found here: `meta/recipes-devtools/buildchroot/buildchroot.bb` > + +This recipe implementes `do_build` task which performs the > following: +1. Generates multistrap config from template: > `meta/recipes-devtools/buildchroot/files/multistrap.conf.in` +2. > Install pre/post scripts for multistrap: > `meta/recipes-devtools/buildchroot/files/configscript.sh` and > `meta/recipes-devtools/buildchroot/files/setup.sh` +3. Run multistrap > +4. Install script for building custom Debian packages: > `meta/recipes-devtools/buildchroot/files/build.sh` + +The single > stamp is created for each user buildchroot configuration. + +### 3.3 > Initial Target Filesystem Generation + +Initial target filesystem > generation process is very similar to buildchroot creating, the > difference is only in initial packages list. + +Target image recipes > are the part of Isar core. There is a sample of typical Isar image > that can be customized according to the user requirements: > `meta-isar/recipes-core/images/isar-image-base.bb` +Like for > buildchroot, the multistrap configuration files for image can be > found here: `meta-isar/recipes-core/images/files`, and it implements > `do_build` task. + +### 3.4 Building Custom Packages + +Isar provides > possibility to build Debian packages from sources. This features > works with Debian-like source packages, i.e. the source code tree > should contain debian folder. The build process is implemented in > `meta/classes/dpkg.bbclass` and consists from the following steps: > +1. Task `do_fetch`: fetch source code from external link +2. Task > `do_unpack`: unpack source code to > `${BUILDCHROOT_DIR}/home/build/${PN}` +3. Task `do_build`: switch to > buildchroot using chroot command and run `build.sh` script. The > `build.sh` script performs the following: > + 1. Go to `/home/build/${PN}` > + 2. Get list of dependencies from debian/control and install them > using apt. > + 3. Run dpkg-buildpackage > +4. Task `do_install`: install successfully built packages > `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy directory > `${DEPLOY_DIR_DEB}` + +### 3.5 Populate Target Filesystem > + > +Each target image can be extended by custom packages listed in > IMAGE_INSTALL variable. Task `do_populate` performs the following: > +1. Parse IMAGE_INSTALL variable +2. Find respective packages in > `${DEPLOY_DIR_DEB}` +3. Copy them to deb folder in dedicated target > filesystem +4. Execute dpkg command in chroot for all the copied > packages + > +### 3.6 Generate Bootable Image > + > +This process contains the following steps: > +1. Task `do_ext4_image`: target filesystem is packed to extfs image > +2. wic tool generates bootable image for dedicated platform