From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6451845424769662976 X-Received: by 10.159.58.231 with SMTP id q39mr3225715uag.21.1502218748865; Tue, 08 Aug 2017 11:59:08 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.107.140.80 with SMTP id o77ls3238547iod.27.gmail; Tue, 08 Aug 2017 11:59:08 -0700 (PDT) X-Received: by 10.129.157.80 with SMTP id u77mr3426597ywg.201.1502218748557; Tue, 08 Aug 2017 11:59:08 -0700 (PDT) Received: by 10.55.212.78 with SMTP id l75msqki; Tue, 8 Aug 2017 08:21:30 -0700 (PDT) X-Received: by 10.28.229.65 with SMTP id c62mr658515wmh.11.1502205690262; Tue, 08 Aug 2017 08:21:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502205690; cv=none; d=google.com; s=arc-20160816; b=rDZFzXgoivSdptlfnxYWhOx6HXehNx3Dp2cNIT/WWP556hcxSKgBVcbszgUMQzBk97 4bQM0JXQYdHwhQZB1Pgk6gsbx6pR2EBYKvhXOaONJNRo6MdxXTsZXzZxBT3W9b3K3fgG 7q4GfzqoxXh+0nljTX4rXAepOL2OnFzdYtsVgAI/QqqK7BqU98Nm/mwZOPIl8EOA6gEP iZs9Ro9Q3BUmF4vDXdVDPQyyLy0FuBbflqVndh0bTE/ZCbQI5YVozSfdbUf9XoIEUwve xHwHZwUtIdBe3RaxbTtDw24jpqFE4InWUeCtZnqoPQFAqSSblN/GAmVKNqGuXnGj9Tuo Lkew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject :arc-authentication-results; bh=YSMd5MMmHSQi2R+OvIOr4zau5peDC7JFDZU7hnY12qg=; b=xEJAg3wFgSvmQoWfL//LRsNLUNfkBZZi6n8kLyJS+ZHOxsaGC+OViFT2hwMwXV6d5y 1jZZ54HZAm58cepvj7zNJV6i959NVB4pR+S+hXzeGANsMwUGxThoQkND35WQuM/e6NXI WEF07/I0x1zRZDgWzVDu2+equrgWY3qaneayiw2IUG473Juu1RIuwJmCaFnIOD6VAeM7 gyvXmMe9yfCS1gLkfMWMCcefM09K77jSpZzOnWU67E0JBuSBidcZdvMK6f91JOJikUhH /53cdQkVdk3ti2XPzjV643O7dZuMN538D6XbJG7fG8H6sZLkKBzmC7Hke8dRi7GNeE92 Qz0A== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id 187si582402wmj.0.2017.08.08.08.21.30 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 08 Aug 2017 08:21:30 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Received: from [10.0.2.15] ([188.227.110.165]) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id v78FLRhf024032 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Tue, 8 Aug 2017 17:21:28 +0200 Subject: Re: [PATCH] doc: Add technical overview To: Henning Schild Cc: isar-users@googlegroups.com References: <20170808100559.19682-1-asmirnov@ilbers.de> <20170808131102.0ef91da0@md1em3qc> <20170808135415.6b88e860@md1em3qc> <247866c8-834d-61d6-7966-bfb702977770@ilbers.de> <20170808150313.55711c49@md1em3qc> <287754c1-1bc6-8fb7-d83c-504adc965a08@ilbers.de> <20170808163845.03fc807c@md1em3qc> <20170808165314.4c7dd289@md1em3qc> From: Alexander Smirnov Message-ID: <8aefc1a4-bafb-23d9-2a80-e65173ae88a2@ilbers.de> Date: Tue, 8 Aug 2017 18:21:22 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: <20170808165314.4c7dd289@md1em3qc> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: TYF19iQjCrMD On 08/08/2017 05:53 PM, Henning Schild wrote: > Am Tue, 8 Aug 2017 17:44:30 +0300 > schrieb Alexander Smirnov : > >> On 08/08/2017 05:38 PM, Henning Schild wrote: >>> Am Tue, 8 Aug 2017 16:12:02 +0300 >>> schrieb Alexander Smirnov : >>> >>>> On 08/08/2017 04:03 PM, Henning Schild wrote: >>>>> Am Tue, 8 Aug 2017 15:41:10 +0300 >>>>> schrieb Alexander Smirnov : >>>>> >>>>>> On 08/08/2017 02:54 PM, Henning Schild wrote: >>>>>>> Am Tue, 8 Aug 2017 14:32:00 +0300 >>>>>>> schrieb Alexander Smirnov : >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> On 08/08/2017 02:11 PM, Henning Schild wrote: >>>>>>>>> I did not really read this yet. At the moment i think we have >>>>>>>>> too many changes in the q to add such a document to the >>>>>>>>> repository. Any change would just mean heaving to keep it up >>>>>>>>> to date. We talked about the need for such a document for the >>>>>>>>> possible OE integration. I think it is valuable for that >>>>>>>>> already. >>>>>>>> >>>>>>>> Briefly speaking, this document describes initial motivation >>>>>>>> and high-level current overview. >>>>>>>> >>>>>>>>> That should be added after we all are happy with the >>>>>>>>> feature-set of Isar, until then it would just be another file >>>>>>>>> to update when changing how things work. >>>>>>>> >>>>>>>> Hmm, this document doesn't affect any discussions about Isar >>>>>>>> feature set. It's intended to be as technical background for >>>>>>>> hot-plug joining to this project. During discussions last >>>>>>>> month I see that we have gap in our understandings how Isar >>>>>>>> works, so this document should be start point to eliminate >>>>>>>> such gaps. >>>>>>>> >>>>>>>> For sure, it's *not* a specification, which explicitly defines >>>>>>>> specific feature set and nothing more will be added. It's just >>>>>>>> an explanation of current state. I see no problems to regularly >>>>>>>> update it, moreover I highly appreciate this. Lack of >>>>>>>> documentation is the most common problem of open source >>>>>>>> projects, so as soon we start documenting the things, the >>>>>>>> easier will be documentation maintenance in future. >>>>>>> >>>>>>> For sure we need that! But especially section 3 contains lots of >>>>>>> interna that will change if my patches get accepted. That is >>>>>>> why i would like to postpone adding the file after my patches >>>>>>> went in. >>>>>> >>>>>> As I said, this document describes current project state, so I >>>>>> don't think that upcoming changes should block publishing of >>>>>> documentation. >>>>>> >>>>>> >>>>>>> >>>>>>> Adding that file before them would mean another rebase round for >>>>>>> documentation. I would be happy to rebase this one for you on >>>>>>> top of a master that contains my stuff, but not too happy about >>>>>>> the other way around. >>>>>> >>>>>> The patch creates document from the scratch, so it can't conflict >>>>>> with any upcoming patches. >>>>> >>>>> It conflicts with other patches that are under review right now. >>>>> All i am asking is to merge conflicting patches in the order the >>>>> patches initially showed up for review, in that case this one >>>>> comes last anyways. And when its time has come it will be >>>>> outdated, and i will update it for you. >>>>> >>>> >>>> Hmm, are you speaking about exactly this patch or about previous >>>> series? >>> >>> Section 3.4 and 3.5 conflict with my changes, they will have to be >>> changed. If your patch comes first i would have to rebase. >>> >>> i.e. >>> - 3.4 talks about dpkg.bbclass which i renamed >>> [PATCH 11-16 of 16 v2 4/6] >>> - 3.5 will not happen anymore if everything enters the rootfs >>> through multistrap >>> [PATCH 3/3] classes: image: remove populate and replace it with a >>> custom repo >> >> I didn't get the idea, how this affects your patches? After your >> features will be applied, you can update this document. In this >> thread you've mentioned, that this patch breaks your series and force >> you to perform rebase, this point is not clear for me. > > A consistent patch updates the code along with the documentation. If > the code i am touching suddenly gets documented my commits will become > inconsistent. You can update the documentation later after your series will be applied. Alex >>> >>> Henning >>> >>> >>>> Alex >>>> >>>>>> Also I expect, that the author of new features will update the >>>>>> documentation, so some reference document should be published >>>>>> anyway. >>>>> >>>>> Sure. >>>>> >>>>>> To be honest I'm not happy with the idea that I'll document all >>>>>> the new features in future. >>>>> >>>>> Missing or inconsistent documentation are a valid reason to not >>>>> accept a patch so it will not come to that. >>>>> >>>>> Henning >>>>> >>>>>> Alex >>>>>> >>>>>>>>> >>>>>>>>> Am Tue, 8 Aug 2017 13:05:59 +0300 >>>>>>>>> schrieb Alexander Smirnov : >>>>>>>>> >>>>>>>>>> Add initial Isar technical overview. >>>>>>>>>> >>>>>>>>>> Signed-off-by: Alexander Smirnov >>>>>>>>>> --- >>>>>>>>>> doc/technical_overview.md | 110 >>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++ 1 file >>>>>>>>>> changed, 110 insertions(+) create mode 100644 >>>>>>>>>> doc/technical_overview.md >>>>>>>>>> >>>>>>>>>> diff --git a/doc/technical_overview.md >>>>>>>>>> b/doc/technical_overview.md new file mode 100644 >>>>>>>>>> index 0000000..f65fc08 >>>>>>>>>> --- /dev/null >>>>>>>>>> +++ b/doc/technical_overview.md >>>>>>>>>> @@ -0,0 +1,110 @@ >>>>>>>>>> +# 1 Introduction >>>>>>>>>> + >>>>>>>>>> +Isar can be shortly described as a set of bitbake recipes >>>>>>>>>> that implement main build system logic. To simplify overall >>>>>>>>>> Isar understanding, this document is split into two main >>>>>>>>>> parts: >>>>>>>>>> + - Isar logical components >>>>>>>>>> + - Isar internal processes >>>>>>>>>> + >>>>>>>>>> +# 2 Isar Logical Components >>>>>>>>>> + >>>>>>>>>> +In this chapter the most important Isar components are >>>>>>>>>> considered in details. In this text component doesn't >>>>>>>>>> especially mean some self-contained single entity of >>>>>>>>>> something, it's just an attempt to split Isar internals by >>>>>>>>>> various criteria. + +### 2.1 Bitbake and Recipes + +All the >>>>>>>>>> processes in Isar are started by bitbake, so it manages >>>>>>>>>> general build executing process. Recipes in Isar can be split >>>>>>>>>> in two categories: + >>>>>>>>>> + - System recipes and classes: they are responsible for >>>>>>>>>> setting up Debian-like infrastructure, manages Debian tools >>>>>>>>>> execution for package building and installation, generation >>>>>>>>>> root file system images >>>>>>>>>> + - User recipes: custom user applications, that should be >>>>>>>>>> builŠµ from sources + >>>>>>>>>> +There are two types of dependencies in Isar: >>>>>>>>>> + - Dependency between bitbake recipes >>>>>>>>>> + - Dependencies in Debian filesystem >>>>>>>>>> + >>>>>>>>>> +**NOTE:** Isar doesn't manage dependencies in Debian file >>>>>>>>>> systems. User can only list specific Debian dependencies in >>>>>>>>>> recipe, so they eventually will be passed to apt-get or >>>>>>>>>> multistrap. Dependency installation is managed by Debian >>>>>>>>>> tools. >>>>>>>>>> + +### 2.2 Stamps + +Each task managed by bitbake uses stamp >>>>>>>>>> to notify that it has been completed. Due to Isar supports >>>>>>>>>> various Debian distributions and parallel builds for >>>>>>>>>> multiple machines and architectures, the stamps for tasks >>>>>>>>>> use special suffixes that include: >>>>>>>>>> + - Debian distro name >>>>>>>>>> + - Architecture >>>>>>>>>> + - Machine >>>>>>>>>> + >>>>>>>>>> +Typical example, when Isar builds the following >>>>>>>>>> configurations: >>>>>>>>>> + - Debian Jessie, amd64 >>>>>>>>>> + - Debian Jessie, i386 >>>>>>>>>> + - Debian Stretch, i386 >>>>>>>>>> + >>>>>>>>>> +In this case there will be 3 different buildchroots, so >>>>>>>>>> standard hello demo application should be processed 3 times >>>>>>>>>> for each environment. Three different sets of stamps should >>>>>>>>>> be used for correct bitbake operating. + +### 2.3 >>>>>>>>>> Buildchroot + +One of the key aspect of Debian philosophy >>>>>>>>>> claims the fact, that everything in Debian should be built >>>>>>>>>> within Debian environment. Moreover native compilation is >>>>>>>>>> more preferable than cross-compilation. To follow this >>>>>>>>>> rules, Isar introduces the new component - buildchroot. >>>>>>>>>> Bulidchroot is typical Debian filesystem that is created >>>>>>>>>> using standard Debian tools: multistrap, apt. The source of >>>>>>>>>> packages can be either official Debian repositories or >>>>>>>>>> custom repositories created by user. + +Buildchroot >>>>>>>>>> lifecycle can be described as following: >>>>>>>>>> + - Buildchroot has initial configuration file which is >>>>>>>>>> passed to multistrap tool. This configuration file is >>>>>>>>>> generated by bitbake recipe from patterns and values defined >>>>>>>>>> by user. Based on this configuration file, multistrap >>>>>>>>>> generates initial filesystem. >>>>>>>>>> + - During building custom Debian package, list of its build >>>>>>>>>> dependencies is installed to buildchroot. >>>>>>>>>> + - When package has been built, it's installed to current >>>>>>>>>> buildchroot to satisfy further packages build dependencies. + >>>>>>>>>> +### 2.4 Target Root Filesystem >>>>>>>>>> + >>>>>>>>>> +Target filesystem is quite similar to buildchroot. The only >>>>>>>>>> difference is that it doesn't have development packages >>>>>>>>>> installed. >>>>>>>>>> + +Target filesystem lifecycle can be described as following: >>>>>>>>>> + - Target filesystem has initial configuration file which is >>>>>>>>>> passed to multistrap tool. This configuration file is >>>>>>>>>> generated by bitbake recipe from patterns and values defined >>>>>>>>>> by user. Based on this configuration file, multistrap >>>>>>>>>> generates initial filesystem. >>>>>>>>>> + - According to the list of custom packages in bitbake >>>>>>>>>> recipes, the initial filesystem will be populated by >>>>>>>>>> successfully built packages. + +# 3 Isar Internal Processes >>>>>>>>>> + >>>>>>>>>> +### 3.1 General Overview >>>>>>>>>> + >>>>>>>>>> +Whole Isar build process can be split into the following >>>>>>>>>> steps: >>>>>>>>>> + - Generation of initial buildchroots for each configuration >>>>>>>>>> (Debian distro, machine and architecture) requested by user. >>>>>>>>>> + - Generation of initial target filesystems for each >>>>>>>>>> configuration. >>>>>>>>>> + - Building custom packages. >>>>>>>>>> + - Populate target filesystems. >>>>>>>>>> + - Generate bootable images. >>>>>>>>>> + >>>>>>>>>> +All these steps are described in details below. >>>>>>>>>> + >>>>>>>>>> +### 3.2 Initial Buildchroot Generation >>>>>>>>>> + >>>>>>>>>> +As mentioned above, initial buildchroot is generated using >>>>>>>>>> multistrap. The bitbake recipe which is responsible for >>>>>>>>>> buildchroot can be found here: >>>>>>>>>> `meta/recipes-devtools/buildchroot/buildchroot.bb` >>>>>>>>>> + +This recipe implementes `do_build` task which performs the >>>>>>>>>> following: +1. Generates multistrap config from template: >>>>>>>>>> `meta/recipes-devtools/buildchroot/files/multistrap.conf.in` >>>>>>>>>> +2. Install pre/post scripts for multistrap: >>>>>>>>>> `meta/recipes-devtools/buildchroot/files/configscript.sh` and >>>>>>>>>> `meta/recipes-devtools/buildchroot/files/setup.sh` +3. Run >>>>>>>>>> multistrap +4. Install script for building custom Debian >>>>>>>>>> packages: `meta/recipes-devtools/buildchroot/files/build.sh` >>>>>>>>>> + +The single stamp is created for each user buildchroot >>>>>>>>>> configuration. + +### 3.3 Initial Target Filesystem >>>>>>>>>> Generation >>>>>>>>>> + +Initial target filesystem generation process is very >>>>>>>>>> similar to buildchroot creating, the difference is only in >>>>>>>>>> initial packages list. + +Target image recipes are the part >>>>>>>>>> of Isar core. There is a sample of typical Isar image that >>>>>>>>>> can be customized according to the user requirements: >>>>>>>>>> `meta-isar/recipes-core/images/isar-image-base.bb` +Like for >>>>>>>>>> buildchroot, the multistrap configuration files for image can >>>>>>>>>> be found here: `meta-isar/recipes-core/images/files`, and it >>>>>>>>>> implements `do_build` task. + +### 3.4 Building Custom >>>>>>>>>> Packages >>>>>>>>>> + +Isar provides possibility to build Debian packages from >>>>>>>>>> sources. This features works with Debian-like source >>>>>>>>>> packages, i.e. the source code tree should contain debian >>>>>>>>>> folder. The build process is implemented in >>>>>>>>>> `meta/classes/dpkg.bbclass` and consists from the following >>>>>>>>>> steps: +1. Task `do_fetch`: fetch source code from external >>>>>>>>>> link +2. Task `do_unpack`: unpack source code to >>>>>>>>>> `${BUILDCHROOT_DIR}/home/build/${PN}` +3. Task `do_build`: >>>>>>>>>> switch to buildchroot using chroot command and run >>>>>>>>>> `build.sh` script. The `build.sh` script performs the >>>>>>>>>> following: >>>>>>>>>> + 1. Go to `/home/build/${PN}` >>>>>>>>>> + 2. Get list of dependencies from debian/control and >>>>>>>>>> install them using apt. >>>>>>>>>> + 3. Run dpkg-buildpackage >>>>>>>>>> +4. Task `do_install`: install successfully built packages >>>>>>>>>> `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy >>>>>>>>>> directory `${DEPLOY_DIR_DEB}` + +### 3.5 Populate Target >>>>>>>>>> Filesystem + >>>>>>>>>> +Each target image can be extended by custom packages listed >>>>>>>>>> in IMAGE_INSTALL variable. Task `do_populate` performs the >>>>>>>>>> following: +1. Parse IMAGE_INSTALL variable +2. Find >>>>>>>>>> respective packages in `${DEPLOY_DIR_DEB}` +3. Copy them to >>>>>>>>>> deb folder in dedicated target filesystem +4. Execute dpkg >>>>>>>>>> command in chroot for all the copied packages + >>>>>>>>>> +### 3.6 Generate Bootable Image >>>>>>>>>> + >>>>>>>>>> +This process contains the following steps: >>>>>>>>>> +1. Task `do_ext4_image`: target filesystem is packed to >>>>>>>>>> extfs image +2. wic tool generates bootable image for >>>>>>>>>> dedicated platform >>>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -- With best regards, Alexander Smirnov ilbers GmbH Baierbrunner Str. 28c D-81379 Munich +49 (89) 122 67 24-0 http://ilbers.de/ Commercial register Munich, HRB 214197 General manager: Baurzhan Ismagulov