From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6451845424769662976 X-Received: by 10.25.81.199 with SMTP id g68mr588041lfl.21.1502203878511; Tue, 08 Aug 2017 07:51:18 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.46.0.218 with SMTP id e87ls720338lji.28.gmail; Tue, 08 Aug 2017 07:51:18 -0700 (PDT) X-Received: by 10.25.217.74 with SMTP id q71mr589332lfg.27.1502203878339; Tue, 08 Aug 2017 07:51:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502203878; cv=none; d=google.com; s=arc-20160816; b=Ohd3b0LLN3SDZviVucnFWqygP5/OBgOHBXo2fv66FlPVH+y+WVmEt2cLdhzpDtubEG flb95Al3jNCayANni5uq7YeL87KbXCjttvHm49M7MZ25DA85uu0i7jPgszTGZsFIhyx5 V2uBgg0gGLgVUYpAjo7rtoFX+snGd8b1ODOQME9Fk6k2HkUHIxVr2Nk5360Y31DnVaSS pOb4vQCKCvnVwDkqsOVtwZlciMFZ/o89Wp2SCSvcT/Ye6RPEsre+ynPf+uGBQcmr8Loc audqs0fd3EGPInX19t2MziIFOMcUa0cwoLSgKt7MvuUtu4XMe4tK8LTSqLYRM+Xvy+6e Oj5A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:arc-authentication-results; bh=Wo0+suLVf566LXVJn8lNoW61xeWxZETVbuU9RZ9pKW8=; b=BhkyG/+MhHd+iDi1in66W1UruTNncEIarmq96YQCJF8NTgstNhUIvmtbj8CwSR+Xcn F9yTVOFP4AtFGRD8AnxNA+S50CGTTh55WbUgtHPwPmI+UZ4umd8yn7ufrgyejIPXacXB apNLN482bjFhCNrAKc3ynObYTxRR9ykQu3Tp2NJtTHkDgdompHF41JgtVwT3bl89Djxo riM1eX6Na//ajWlPHa1eqsfTTmuuJTCrfIlI3zRVNResAqzlLwcarJ353KM2dIOx+2Ys Uv/tqLVRioSdZBxAaqYqgSHICavH1ONvXi8D6MW1VkO5sUIeNu7DdKInkBdBmWGXun1T N7UA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 192.35.17.2 is neither permitted nor denied by best guess record for domain of henning.schild@siemens.com) smtp.mailfrom=henning.schild@siemens.com Return-Path: Received: from thoth.sbs.de (thoth.sbs.de. [192.35.17.2]) by gmr-mx.google.com with ESMTPS id 81si546210wmh.1.2017.08.08.07.51.18 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 08 Aug 2017 07:51:18 -0700 (PDT) Received-SPF: neutral (google.com: 192.35.17.2 is neither permitted nor denied by best guess record for domain of henning.schild@siemens.com) client-ip=192.35.17.2; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 192.35.17.2 is neither permitted nor denied by best guess record for domain of henning.schild@siemens.com) smtp.mailfrom=henning.schild@siemens.com Received: from mail3.siemens.de (mail3.siemens.de [139.25.208.14]) by thoth.sbs.de (8.15.2/8.15.2) with ESMTPS id v78EpHFn008911 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 8 Aug 2017 16:51:18 +0200 Received: from md1em3qc ([139.25.68.40]) by mail3.siemens.de (8.15.2/8.15.2) with ESMTP id v78EpHHl025576; Tue, 8 Aug 2017 16:51:17 +0200 Date: Tue, 8 Aug 2017 16:53:14 +0200 From: Henning Schild To: Alexander Smirnov Cc: Subject: Re: [PATCH] doc: Add technical overview Message-ID: <20170808165314.4c7dd289@md1em3qc> In-Reply-To: References: <20170808100559.19682-1-asmirnov@ilbers.de> <20170808131102.0ef91da0@md1em3qc> <20170808135415.6b88e860@md1em3qc> <247866c8-834d-61d6-7966-bfb702977770@ilbers.de> <20170808150313.55711c49@md1em3qc> <287754c1-1bc6-8fb7-d83c-504adc965a08@ilbers.de> <20170808163845.03fc807c@md1em3qc> X-Mailer: Claws Mail 3.13.2 (GTK+ 2.24.31; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-TUID: STkN8eCFHWzE Am Tue, 8 Aug 2017 17:44:30 +0300 schrieb Alexander Smirnov : > On 08/08/2017 05:38 PM, Henning Schild wrote: > > Am Tue, 8 Aug 2017 16:12:02 +0300 > > schrieb Alexander Smirnov : > > =20 > >> On 08/08/2017 04:03 PM, Henning Schild wrote: =20 > >>> Am Tue, 8 Aug 2017 15:41:10 +0300 > >>> schrieb Alexander Smirnov : > >>> =20 > >>>> On 08/08/2017 02:54 PM, Henning Schild wrote: =20 > >>>>> Am Tue, 8 Aug 2017 14:32:00 +0300 > >>>>> schrieb Alexander Smirnov : > >>>>> =20 > >>>>>> Hi, > >>>>>> > >>>>>> On 08/08/2017 02:11 PM, Henning Schild wrote: =20 > >>>>>>> I did not really read this yet. At the moment i think we have > >>>>>>> too many changes in the q to add such a document to the > >>>>>>> repository. Any change would just mean heaving to keep it up > >>>>>>> to date. We talked about the need for such a document for the > >>>>>>> possible OE integration. I think it is valuable for that > >>>>>>> already. =20 > >>>>>> > >>>>>> Briefly speaking, this document describes initial motivation > >>>>>> and high-level current overview. > >>>>>> =20 > >>>>>>> That should be added after we all are happy with the > >>>>>>> feature-set of Isar, until then it would just be another file > >>>>>>> to update when changing how things work. =20 > >>>>>> > >>>>>> Hmm, this document doesn't affect any discussions about Isar > >>>>>> feature set. It's intended to be as technical background for > >>>>>> hot-plug joining to this project. During discussions last > >>>>>> month I see that we have gap in our understandings how Isar > >>>>>> works, so this document should be start point to eliminate > >>>>>> such gaps. > >>>>>> > >>>>>> For sure, it's *not* a specification, which explicitly defines > >>>>>> specific feature set and nothing more will be added. It's just > >>>>>> an explanation of current state. I see no problems to regularly > >>>>>> update it, moreover I highly appreciate this. Lack of > >>>>>> documentation is the most common problem of open source > >>>>>> projects, so as soon we start documenting the things, the > >>>>>> easier will be documentation maintenance in future. =20 > >>>>> > >>>>> For sure we need that! But especially section 3 contains lots of > >>>>> interna that will change if my patches get accepted. That is > >>>>> why i would like to postpone adding the file after my patches > >>>>> went in. =20 > >>>> > >>>> As I said, this document describes current project state, so I > >>>> don't think that upcoming changes should block publishing of > >>>> documentation. > >>>> > >>>> =20 > >>>>> > >>>>> Adding that file before them would mean another rebase round for > >>>>> documentation. I would be happy to rebase this one for you on > >>>>> top of a master that contains my stuff, but not too happy about > >>>>> the other way around. =20 > >>>> > >>>> The patch creates document from the scratch, so it can't conflict > >>>> with any upcoming patches. =20 > >>> > >>> It conflicts with other patches that are under review right now. > >>> All i am asking is to merge conflicting patches in the order the > >>> patches initially showed up for review, in that case this one > >>> comes last anyways. And when its time has come it will be > >>> outdated, and i will update it for you. > >>> =20 > >> > >> Hmm, are you speaking about exactly this patch or about previous > >> series? =20 > >=20 > > Section 3.4 and 3.5 conflict with my changes, they will have to be > > changed. If your patch comes first i would have to rebase. > >=20 > > i.e. > > - 3.4 talks about dpkg.bbclass which i renamed > > [PATCH 11-16 of 16 v2 4/6] > > - 3.5 will not happen anymore if everything enters the rootfs > > through multistrap > > [PATCH 3/3] classes: image: remove populate and replace it with a > > custom repo =20 >=20 > I didn't get the idea, how this affects your patches? After your=20 > features will be applied, you can update this document. In this > thread you've mentioned, that this patch breaks your series and force > you to perform rebase, this point is not clear for me. A consistent patch updates the code along with the documentation. If the code i am touching suddenly gets documented my commits will become inconsistent. Henning > >=20 > >=20 > > Henning > >=20 > > =20 > >> Alex > >> =20 > >>>> Also I expect, that the author of new features will update the > >>>> documentation, so some reference document should be published > >>>> anyway. =20 > >>> > >>> Sure. > >>> =20 > >>>> To be honest I'm not happy with the idea that I'll document all > >>>> the new features in future. =20 > >>> > >>> Missing or inconsistent documentation are a valid reason to not > >>> accept a patch so it will not come to that. > >>> > >>> Henning > >>> =20 > >>>> Alex > >>>> =20 > >>>>>>> > >>>>>>> Am Tue, 8 Aug 2017 13:05:59 +0300 > >>>>>>> schrieb Alexander Smirnov : > >>>>>>> =20 > >>>>>>>> Add initial Isar technical overview. > >>>>>>>> > >>>>>>>> Signed-off-by: Alexander Smirnov > >>>>>>>> --- > >>>>>>>> doc/technical_overview.md | 110 > >>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++ 1 file > >>>>>>>> changed, 110 insertions(+) create mode 100644 > >>>>>>>> doc/technical_overview.md > >>>>>>>> > >>>>>>>> diff --git a/doc/technical_overview.md > >>>>>>>> b/doc/technical_overview.md new file mode 100644 > >>>>>>>> index 0000000..f65fc08 > >>>>>>>> --- /dev/null > >>>>>>>> +++ b/doc/technical_overview.md > >>>>>>>> @@ -0,0 +1,110 @@ > >>>>>>>> +# 1 Introduction > >>>>>>>> + > >>>>>>>> +Isar can be shortly described as a set of bitbake recipes > >>>>>>>> that implement main build system logic. To simplify overall > >>>>>>>> Isar understanding, this document is split into two main > >>>>>>>> parts: > >>>>>>>> + - Isar logical components > >>>>>>>> + - Isar internal processes > >>>>>>>> + > >>>>>>>> +# 2 Isar Logical Components > >>>>>>>> + > >>>>>>>> +In this chapter the most important Isar components are > >>>>>>>> considered in details. In this text component doesn't > >>>>>>>> especially mean some self-contained single entity of > >>>>>>>> something, it's just an attempt to split Isar internals by > >>>>>>>> various criteria. + +### 2.1 Bitbake and Recipes + +All the > >>>>>>>> processes in Isar are started by bitbake, so it manages > >>>>>>>> general build executing process. Recipes in Isar can be split > >>>>>>>> in two categories: + > >>>>>>>> + - System recipes and classes: they are responsible for > >>>>>>>> setting up Debian-like infrastructure, manages Debian tools > >>>>>>>> execution for package building and installation, generation > >>>>>>>> root file system images > >>>>>>>> + - User recipes: custom user applications, that should be > >>>>>>>> buil=D0=B5 from sources + > >>>>>>>> +There are two types of dependencies in Isar: > >>>>>>>> + - Dependency between bitbake recipes > >>>>>>>> + - Dependencies in Debian filesystem > >>>>>>>> + > >>>>>>>> +**NOTE:** Isar doesn't manage dependencies in Debian file > >>>>>>>> systems. User can only list specific Debian dependencies in > >>>>>>>> recipe, so they eventually will be passed to apt-get or > >>>>>>>> multistrap. Dependency installation is managed by Debian > >>>>>>>> tools. > >>>>>>>> + +### 2.2 Stamps + +Each task managed by bitbake uses stamp > >>>>>>>> to notify that it has been completed. Due to Isar supports > >>>>>>>> various Debian distributions and parallel builds for > >>>>>>>> multiple machines and architectures, the stamps for tasks > >>>>>>>> use special suffixes that include: > >>>>>>>> + - Debian distro name > >>>>>>>> + - Architecture > >>>>>>>> + - Machine > >>>>>>>> + > >>>>>>>> +Typical example, when Isar builds the following > >>>>>>>> configurations: > >>>>>>>> + - Debian Jessie, amd64 > >>>>>>>> + - Debian Jessie, i386 > >>>>>>>> + - Debian Stretch, i386 > >>>>>>>> + > >>>>>>>> +In this case there will be 3 different buildchroots, so > >>>>>>>> standard hello demo application should be processed 3 times > >>>>>>>> for each environment. Three different sets of stamps should > >>>>>>>> be used for correct bitbake operating. + +### 2.3 > >>>>>>>> Buildchroot + +One of the key aspect of Debian philosophy > >>>>>>>> claims the fact, that everything in Debian should be built > >>>>>>>> within Debian environment. Moreover native compilation is > >>>>>>>> more preferable than cross-compilation. To follow this > >>>>>>>> rules, Isar introduces the new component - buildchroot. > >>>>>>>> Bulidchroot is typical Debian filesystem that is created > >>>>>>>> using standard Debian tools: multistrap, apt. The source of > >>>>>>>> packages can be either official Debian repositories or > >>>>>>>> custom repositories created by user. + +Buildchroot > >>>>>>>> lifecycle can be described as following: > >>>>>>>> + - Buildchroot has initial configuration file which is > >>>>>>>> passed to multistrap tool. This configuration file is > >>>>>>>> generated by bitbake recipe from patterns and values defined > >>>>>>>> by user. Based on this configuration file, multistrap > >>>>>>>> generates initial filesystem. > >>>>>>>> + - During building custom Debian package, list of its build > >>>>>>>> dependencies is installed to buildchroot. > >>>>>>>> + - When package has been built, it's installed to current > >>>>>>>> buildchroot to satisfy further packages build dependencies. + > >>>>>>>> +### 2.4 Target Root Filesystem > >>>>>>>> + > >>>>>>>> +Target filesystem is quite similar to buildchroot. The only > >>>>>>>> difference is that it doesn't have development packages > >>>>>>>> installed. > >>>>>>>> + +Target filesystem lifecycle can be described as following: > >>>>>>>> + - Target filesystem has initial configuration file which is > >>>>>>>> passed to multistrap tool. This configuration file is > >>>>>>>> generated by bitbake recipe from patterns and values defined > >>>>>>>> by user. Based on this configuration file, multistrap > >>>>>>>> generates initial filesystem. > >>>>>>>> + - According to the list of custom packages in bitbake > >>>>>>>> recipes, the initial filesystem will be populated by > >>>>>>>> successfully built packages. + +# 3 Isar Internal Processes > >>>>>>>> + > >>>>>>>> +### 3.1 General Overview > >>>>>>>> + > >>>>>>>> +Whole Isar build process can be split into the following > >>>>>>>> steps: > >>>>>>>> + - Generation of initial buildchroots for each configuration > >>>>>>>> (Debian distro, machine and architecture) requested by user. > >>>>>>>> + - Generation of initial target filesystems for each > >>>>>>>> configuration. > >>>>>>>> + - Building custom packages. > >>>>>>>> + - Populate target filesystems. > >>>>>>>> + - Generate bootable images. > >>>>>>>> + > >>>>>>>> +All these steps are described in details below. > >>>>>>>> + > >>>>>>>> +### 3.2 Initial Buildchroot Generation > >>>>>>>> + > >>>>>>>> +As mentioned above, initial buildchroot is generated using > >>>>>>>> multistrap. The bitbake recipe which is responsible for > >>>>>>>> buildchroot can be found here: > >>>>>>>> `meta/recipes-devtools/buildchroot/buildchroot.bb` > >>>>>>>> + +This recipe implementes `do_build` task which performs the > >>>>>>>> following: +1. Generates multistrap config from template: > >>>>>>>> `meta/recipes-devtools/buildchroot/files/multistrap.conf.in` > >>>>>>>> +2. Install pre/post scripts for multistrap: > >>>>>>>> `meta/recipes-devtools/buildchroot/files/configscript.sh` and > >>>>>>>> `meta/recipes-devtools/buildchroot/files/setup.sh` +3. Run > >>>>>>>> multistrap +4. Install script for building custom Debian > >>>>>>>> packages: `meta/recipes-devtools/buildchroot/files/build.sh` > >>>>>>>> + +The single stamp is created for each user buildchroot > >>>>>>>> configuration. + +### 3.3 Initial Target Filesystem > >>>>>>>> Generation > >>>>>>>> + +Initial target filesystem generation process is very > >>>>>>>> similar to buildchroot creating, the difference is only in > >>>>>>>> initial packages list. + +Target image recipes are the part > >>>>>>>> of Isar core. There is a sample of typical Isar image that > >>>>>>>> can be customized according to the user requirements: > >>>>>>>> `meta-isar/recipes-core/images/isar-image-base.bb` +Like for > >>>>>>>> buildchroot, the multistrap configuration files for image can > >>>>>>>> be found here: `meta-isar/recipes-core/images/files`, and it > >>>>>>>> implements `do_build` task. + +### 3.4 Building Custom > >>>>>>>> Packages > >>>>>>>> + +Isar provides possibility to build Debian packages from > >>>>>>>> sources. This features works with Debian-like source > >>>>>>>> packages, i.e. the source code tree should contain debian > >>>>>>>> folder. The build process is implemented in > >>>>>>>> `meta/classes/dpkg.bbclass` and consists from the following > >>>>>>>> steps: +1. Task `do_fetch`: fetch source code from external > >>>>>>>> link +2. Task `do_unpack`: unpack source code to > >>>>>>>> `${BUILDCHROOT_DIR}/home/build/${PN}` +3. Task `do_build`: > >>>>>>>> switch to buildchroot using chroot command and run > >>>>>>>> `build.sh` script. The `build.sh` script performs the > >>>>>>>> following: > >>>>>>>> + 1. Go to `/home/build/${PN}` > >>>>>>>> + 2. Get list of dependencies from debian/control and > >>>>>>>> install them using apt. > >>>>>>>> + 3. Run dpkg-buildpackage > >>>>>>>> +4. Task `do_install`: install successfully built packages > >>>>>>>> `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy > >>>>>>>> directory `${DEPLOY_DIR_DEB}` + +### 3.5 Populate Target > >>>>>>>> Filesystem + > >>>>>>>> +Each target image can be extended by custom packages listed > >>>>>>>> in IMAGE_INSTALL variable. Task `do_populate` performs the > >>>>>>>> following: +1. Parse IMAGE_INSTALL variable +2. Find > >>>>>>>> respective packages in `${DEPLOY_DIR_DEB}` +3. Copy them to > >>>>>>>> deb folder in dedicated target filesystem +4. Execute dpkg > >>>>>>>> command in chroot for all the copied packages + > >>>>>>>> +### 3.6 Generate Bootable Image > >>>>>>>> + > >>>>>>>> +This process contains the following steps: > >>>>>>>> +1. Task `do_ext4_image`: target filesystem is packed to > >>>>>>>> extfs image +2. wic tool generates bootable image for > >>>>>>>> dedicated platform =20 > >>>>>>> =20 > >>>>> =20 > >>>> =20 > >>> =20 > >> =20 > > =20 >=20