From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6451845424769662976 X-Received: by 10.223.143.3 with SMTP id p3mr362176wrb.10.1502193139547; Tue, 08 Aug 2017 04:52:19 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.211.210 with SMTP id k201ls699851wmg.25.gmail; Tue, 08 Aug 2017 04:52:19 -0700 (PDT) X-Received: by 10.28.156.129 with SMTP id f123mr563595wme.19.1502193139151; Tue, 08 Aug 2017 04:52:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502193139; cv=none; d=google.com; s=arc-20160816; b=IPMjtu/D/IMcWsqPB7fZbImNPKAUmaX0s/15KrbXbff+p8pe8XmdSV6qFUhiSApFc0 hN00Wk2XdSaiTm5S/CuU91DcNxlCgVz0Jnnc8ZXPgj9of9j0f2sfEQZm+FM5qLzKHslm T+ok05pzeIZzWHM51cGZT8fp3tk6MI7jbioCr8Afco2EqZl2J61eOMCGotRduacod9Hn MzzhifnqhWO1AgOjAckO3HsnPtmocDMXEoY34p30BuqLyxENlSBH2hk9rrqvfRcprb8B mhVbyJDeDoTa2f335c8StQEByd43FgFhiHXcMHN87Q6OOUuCnVAEEloPtijMJw19zwzk /cmQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:arc-authentication-results; bh=vwifj+er5C1EQP89p8cDwjdFBqsHbavh3Tb7zeNsbMM=; b=MLwwUChWErjvspXTuUl/0a8LF1cdC45HxZ87v0SltkSrRiGrFXjyw7uvGUcxIT/0q4 xP3FN8Df7BykuN/dHtVOHV3/7L/6Vu8xkWC0CaTUGaZeG8+OnsmESffHKaFI5ChRbXTx hvCOM+9ZG7exPa0Y6PRYE4tpYeNKjwcbxNIXFNtgQrQHX93ZhhBLdh3kQv8vFympJ1sT Tx4Y25GJHh2GP3Gd1aKkktFOkUswiPR+40+6CsS36toLvWt5CIcClVJrlBsgoyNxCz6S PC+0ev8SLQ/IMeq1oQeZFBBB5U2tGx/xT5A/TpwcHze6OqVT+3U7lZ4Poy8IyuUTML8h nIUg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 192.35.17.2 is neither permitted nor denied by best guess record for domain of henning.schild@siemens.com) smtp.mailfrom=henning.schild@siemens.com Return-Path: Received: from thoth.sbs.de (thoth.sbs.de. [192.35.17.2]) by gmr-mx.google.com with ESMTPS id v142si412994wmv.8.2017.08.08.04.52.19 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 08 Aug 2017 04:52:19 -0700 (PDT) Received-SPF: neutral (google.com: 192.35.17.2 is neither permitted nor denied by best guess record for domain of henning.schild@siemens.com) client-ip=192.35.17.2; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 192.35.17.2 is neither permitted nor denied by best guess record for domain of henning.schild@siemens.com) smtp.mailfrom=henning.schild@siemens.com Received: from mail3.siemens.de (mail3.siemens.de [139.25.208.14]) by thoth.sbs.de (8.15.2/8.15.2) with ESMTPS id v78BqIPh030873 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 8 Aug 2017 13:52:18 +0200 Received: from md1em3qc ([139.25.68.40]) by mail3.siemens.de (8.15.2/8.15.2) with ESMTP id v78BqISH018803; Tue, 8 Aug 2017 13:52:18 +0200 Date: Tue, 8 Aug 2017 13:54:15 +0200 From: Henning Schild To: Alexander Smirnov Cc: Subject: Re: [PATCH] doc: Add technical overview Message-ID: <20170808135415.6b88e860@md1em3qc> In-Reply-To: References: <20170808100559.19682-1-asmirnov@ilbers.de> <20170808131102.0ef91da0@md1em3qc> X-Mailer: Claws Mail 3.13.2 (GTK+ 2.24.31; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-TUID: ZBQaGYOL/g8U Am Tue, 8 Aug 2017 14:32:00 +0300 schrieb Alexander Smirnov : > Hi, >=20 > On 08/08/2017 02:11 PM, Henning Schild wrote: > > I did not really read this yet. At the moment i think we have too > > many changes in the q to add such a document to the repository. Any > > change would just mean heaving to keep it up to date. > > We talked about the need for such a document for the possible OE > > integration. I think it is valuable for that already. > > =20 >=20 > Briefly speaking, this document describes initial motivation and=20 > high-level current overview. >=20 > > That should be added after we all are happy with the feature-set of > > Isar, until then it would just be another file to update when > > changing how things work. =20 >=20 > Hmm, this document doesn't affect any discussions about Isar feature=20 > set. It's intended to be as technical background for hot-plug joining > to this project. During discussions last month I see that we have gap > in our understandings how Isar works, so this document should be > start point to eliminate such gaps. >=20 > For sure, it's *not* a specification, which explicitly defines > specific feature set and nothing more will be added. It's just an > explanation of current state. I see no problems to regularly update > it, moreover I highly appreciate this. Lack of documentation is the > most common problem of open source projects, so as soon we start > documenting the things, the easier will be documentation maintenance > in future. For sure we need that! But especially section 3 contains lots of interna that will change if my patches get accepted. That is why i would like to postpone adding the file after my patches went in.=20 Adding that file before them would mean another rebase round for documentation. I would be happy to rebase this one for you on top of a master that contains my stuff, but not too happy about the other way around. Henning > Alex >=20 > >=20 > > Am Tue, 8 Aug 2017 13:05:59 +0300 > > schrieb Alexander Smirnov : > > =20 > >> Add initial Isar technical overview. > >> > >> Signed-off-by: Alexander Smirnov > >> --- > >> doc/technical_overview.md | 110 > >> ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 110 > >> insertions(+) create mode 100644 doc/technical_overview.md > >> > >> diff --git a/doc/technical_overview.md b/doc/technical_overview.md > >> new file mode 100644 > >> index 0000000..f65fc08 > >> --- /dev/null > >> +++ b/doc/technical_overview.md > >> @@ -0,0 +1,110 @@ > >> +# 1 Introduction > >> + > >> +Isar can be shortly described as a set of bitbake recipes that > >> implement main build system logic. To simplify overall Isar > >> understanding, this document is split into two main parts: > >> + - Isar logical components > >> + - Isar internal processes > >> + > >> +# 2 Isar Logical Components > >> + > >> +In this chapter the most important Isar components are considered > >> in details. In this text component doesn't especially mean some > >> self-contained single entity of something, it's just an attempt to > >> split Isar internals by various criteria. + +### 2.1 Bitbake and > >> Recipes + +All the processes in Isar are started by bitbake, so it > >> manages general build executing process. Recipes in Isar can be > >> split in two categories: + > >> + - System recipes and classes: they are responsible for setting up > >> Debian-like infrastructure, manages Debian tools execution for > >> package building and installation, generation root file system > >> images > >> + - User recipes: custom user applications, that should be buil=D0=B5 > >> from sources + > >> +There are two types of dependencies in Isar: > >> + - Dependency between bitbake recipes > >> + - Dependencies in Debian filesystem > >> + > >> +**NOTE:** Isar doesn't manage dependencies in Debian file systems. > >> User can only list specific Debian dependencies in recipe, so they > >> eventually will be passed to apt-get or multistrap. Dependency > >> installation is managed by Debian tools. + +### 2.2 Stamps + > >> +Each task managed by bitbake uses stamp to notify that it has been > >> completed. Due to Isar supports various Debian distributions and > >> parallel builds for multiple machines and architectures, the stamps > >> for tasks use special suffixes that include: > >> + - Debian distro name > >> + - Architecture > >> + - Machine > >> + > >> +Typical example, when Isar builds the following configurations: > >> + - Debian Jessie, amd64 > >> + - Debian Jessie, i386 > >> + - Debian Stretch, i386 > >> + > >> +In this case there will be 3 different buildchroots, so standard > >> hello demo application should be processed 3 times for each > >> environment. Three different sets of stamps should be used for > >> correct bitbake operating. + +### 2.3 Buildchroot + > >> +One of the key aspect of Debian philosophy claims the fact, that > >> everything in Debian should be built within Debian environment. > >> Moreover native compilation is more preferable than > >> cross-compilation. To follow this rules, Isar introduces the new > >> component - buildchroot. Bulidchroot is typical Debian filesystem > >> that is created using standard Debian tools: multistrap, apt. The > >> source of packages can be either official Debian repositories or > >> custom repositories created by user. + +Buildchroot lifecycle can > >> be described as following: > >> + - Buildchroot has initial configuration file which is passed to > >> multistrap tool. This configuration file is generated by bitbake > >> recipe from patterns and values defined by user. Based on this > >> configuration file, multistrap generates initial filesystem. > >> + - During building custom Debian package, list of its build > >> dependencies is installed to buildchroot. > >> + - When package has been built, it's installed to current > >> buildchroot to satisfy further packages build dependencies. + > >> +### 2.4 Target Root Filesystem > >> + > >> +Target filesystem is quite similar to buildchroot. The only > >> difference is that it doesn't have development packages installed. > >> + +Target filesystem lifecycle can be described as following: > >> + - Target filesystem has initial configuration file which is > >> passed to multistrap tool. This configuration file is generated by > >> bitbake recipe from patterns and values defined by user. Based on > >> this configuration file, multistrap generates initial filesystem. > >> + - According to the list of custom packages in bitbake recipes, > >> the initial filesystem will be populated by successfully built > >> packages. + +# 3 Isar Internal Processes > >> + > >> +### 3.1 General Overview > >> + > >> +Whole Isar build process can be split into the following steps: > >> + - Generation of initial buildchroots for each configuration > >> (Debian distro, machine and architecture) requested by user. > >> + - Generation of initial target filesystems for each > >> configuration. > >> + - Building custom packages. > >> + - Populate target filesystems. > >> + - Generate bootable images. > >> + > >> +All these steps are described in details below. > >> + > >> +### 3.2 Initial Buildchroot Generation > >> + > >> +As mentioned above, initial buildchroot is generated using > >> multistrap. The bitbake recipe which is responsible for buildchroot > >> can be found here: > >> `meta/recipes-devtools/buildchroot/buildchroot.bb` > >> + +This recipe implementes `do_build` task which performs the > >> following: +1. Generates multistrap config from template: > >> `meta/recipes-devtools/buildchroot/files/multistrap.conf.in` +2. > >> Install pre/post scripts for multistrap: > >> `meta/recipes-devtools/buildchroot/files/configscript.sh` and > >> `meta/recipes-devtools/buildchroot/files/setup.sh` +3. Run > >> multistrap +4. Install script for building custom Debian packages: > >> `meta/recipes-devtools/buildchroot/files/build.sh` + +The single > >> stamp is created for each user buildchroot configuration. + +### > >> 3.3 Initial Target Filesystem Generation + +Initial target > >> filesystem generation process is very similar to buildchroot > >> creating, the difference is only in initial packages list. + > >> +Target image recipes are the part of Isar core. There is a sample > >> of typical Isar image that can be customized according to the user > >> requirements: `meta-isar/recipes-core/images/isar-image-base.bb` > >> +Like for buildchroot, the multistrap configuration files for > >> image can be found here: `meta-isar/recipes-core/images/files`, > >> and it implements `do_build` task. + +### 3.4 Building Custom > >> Packages + +Isar provides possibility to build Debian packages > >> from sources. This features works with Debian-like source > >> packages, i.e. the source code tree should contain debian folder. > >> The build process is implemented in `meta/classes/dpkg.bbclass` > >> and consists from the following steps: +1. Task `do_fetch`: fetch > >> source code from external link +2. Task `do_unpack`: unpack source > >> code to `${BUILDCHROOT_DIR}/home/build/${PN}` +3. Task `do_build`: > >> switch to buildchroot using chroot command and run `build.sh` > >> script. The `build.sh` script performs the following: > >> + 1. Go to `/home/build/${PN}` > >> + 2. Get list of dependencies from debian/control and install > >> them using apt. > >> + 3. Run dpkg-buildpackage > >> +4. Task `do_install`: install successfully built packages > >> `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy directory > >> `${DEPLOY_DIR_DEB}` + +### 3.5 Populate Target Filesystem > >> + > >> +Each target image can be extended by custom packages listed in > >> IMAGE_INSTALL variable. Task `do_populate` performs the following: > >> +1. Parse IMAGE_INSTALL variable +2. Find respective packages in > >> `${DEPLOY_DIR_DEB}` +3. Copy them to deb folder in dedicated target > >> filesystem +4. Execute dpkg command in chroot for all the copied > >> packages + > >> +### 3.6 Generate Bootable Image > >> + > >> +This process contains the following steps: > >> +1. Task `do_ext4_image`: target filesystem is packed to extfs > >> image +2. wic tool generates bootable image for dedicated > >> platform =20 > > =20