From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6451845424769662976 X-Received: by 10.99.177.1 with SMTP id r1mr828868pgf.2.1502218748954; Tue, 08 Aug 2017 11:59:08 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.107.179.139 with SMTP id c133ls9286453iof.36.gmail; Tue, 08 Aug 2017 11:59:08 -0700 (PDT) X-Received: by 10.176.17.228 with SMTP id q36mr3145177uac.56.1502218748675; Tue, 08 Aug 2017 11:59:08 -0700 (PDT) Received: by 10.55.163.202 with SMTP id m193msqke; Tue, 8 Aug 2017 06:12:11 -0700 (PDT) X-Received: by 10.28.96.194 with SMTP id u185mr561121wmb.26.1502197931317; Tue, 08 Aug 2017 06:12:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502197931; cv=none; d=google.com; s=arc-20160816; b=J3ES+RAK6Rf/caHrQkWueHiQ9IgJ2xLvZ6Ib4YwSNrU/PLfQXB9zAjAcHkIhhkSU49 wdt60bFwqv963LRwBRzxjYjCoCmoNhMrGqxmP/XKlkxOtpwm+8r4CRZaiR+wS1HvK5DK mhOBJrfO0fuIpZuk1Ho3jAjqhWoGLfaz4Vja9cIPH3SDcC5AnqaTXgF80LGYMCU71dW9 G4/KC9iHjQfuikgjaN4bmbDrWlrlytJ8EKuiXWOmP2sz1cQ8GutNklLJHcKqlM1QBYFk Uxi+xiWpX36s2ZvkhYHUngOlFq/A6eV3TaWIGa/viDRxE63jojosXIivaladRZEqNvf7 5Nwg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject :arc-authentication-results; bh=57oGyM9yyHq4wYQKYqhgeMlwQx3yrV6pCteX5LcLZ+I=; b=SC18S8bOMuytzAD+xHwSG5JCSwnG1V2Qx4TYVzHgDBMTxL1AYO+BHKoyrf9ErZcDb5 ddoPXe0f3j44WpbF0nEiPnVMDiQvZaEVmFqwOSoRn6f7WSh0cMQSoTOzvZS8+U4avlTa Cv/6K2nASC3BW/lHqlRBkGtEFok+WZ2Qncakt84zq44ImrtO6N0uH2eZ+AerFpxDFBwa CZwi7ZYNp/m93UmTlBBbyVztIpp8y7ADJufURj7RHvpYHkB41bMpUKg5CGGUTbIdf1E3 kUNOTMrF+ytc9fAWi3YP+7cZivsaGP4v3skPR38e0Wca/fhRNJEclmFaLO2R7akgH5KT dIxw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id r83si449787wma.9.2017.08.08.06.12.11 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 08 Aug 2017 06:12:11 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Received: from [10.0.2.15] ([188.227.110.165]) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id v78DC7UU022971 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Tue, 8 Aug 2017 15:12:09 +0200 Subject: Re: [PATCH] doc: Add technical overview To: Henning Schild Cc: isar-users@googlegroups.com References: <20170808100559.19682-1-asmirnov@ilbers.de> <20170808131102.0ef91da0@md1em3qc> <20170808135415.6b88e860@md1em3qc> <247866c8-834d-61d6-7966-bfb702977770@ilbers.de> <20170808150313.55711c49@md1em3qc> From: Alexander Smirnov Message-ID: <287754c1-1bc6-8fb7-d83c-504adc965a08@ilbers.de> Date: Tue, 8 Aug 2017 16:12:02 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: <20170808150313.55711c49@md1em3qc> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: uNq5uA7+xxs/ On 08/08/2017 04:03 PM, Henning Schild wrote: > Am Tue, 8 Aug 2017 15:41:10 +0300 > schrieb Alexander Smirnov : > >> On 08/08/2017 02:54 PM, Henning Schild wrote: >>> Am Tue, 8 Aug 2017 14:32:00 +0300 >>> schrieb Alexander Smirnov : >>> >>>> Hi, >>>> >>>> On 08/08/2017 02:11 PM, Henning Schild wrote: >>>>> I did not really read this yet. At the moment i think we have too >>>>> many changes in the q to add such a document to the repository. >>>>> Any change would just mean heaving to keep it up to date. >>>>> We talked about the need for such a document for the possible OE >>>>> integration. I think it is valuable for that already. >>>>> >>>> >>>> Briefly speaking, this document describes initial motivation and >>>> high-level current overview. >>>> >>>>> That should be added after we all are happy with the feature-set >>>>> of Isar, until then it would just be another file to update when >>>>> changing how things work. >>>> >>>> Hmm, this document doesn't affect any discussions about Isar >>>> feature set. It's intended to be as technical background for >>>> hot-plug joining to this project. During discussions last month I >>>> see that we have gap in our understandings how Isar works, so this >>>> document should be start point to eliminate such gaps. >>>> >>>> For sure, it's *not* a specification, which explicitly defines >>>> specific feature set and nothing more will be added. It's just an >>>> explanation of current state. I see no problems to regularly update >>>> it, moreover I highly appreciate this. Lack of documentation is the >>>> most common problem of open source projects, so as soon we start >>>> documenting the things, the easier will be documentation >>>> maintenance in future. >>> >>> For sure we need that! But especially section 3 contains lots of >>> interna that will change if my patches get accepted. That is why i >>> would like to postpone adding the file after my patches went in. >> >> As I said, this document describes current project state, so I don't >> think that upcoming changes should block publishing of documentation. >> >> >>> >>> Adding that file before them would mean another rebase round for >>> documentation. I would be happy to rebase this one for you on top >>> of a master that contains my stuff, but not too happy about the >>> other way around. >> >> The patch creates document from the scratch, so it can't conflict >> with any upcoming patches. > > It conflicts with other patches that are under review right now. All i > am asking is to merge conflicting patches in the order the patches > initially showed up for review, in that case this one comes last > anyways. And when its time has come it will be outdated, and i will > update it for you. > Hmm, are you speaking about exactly this patch or about previous series? Alex >> Also I expect, that the author of new features will update the >> documentation, so some reference document should be published anyway. > > Sure. > >> To be honest I'm not happy with the idea that I'll document all the >> new features in future. > > Missing or inconsistent documentation are a valid reason to not accept > a patch so it will not come to that. > > Henning > >> Alex >> >>>>> >>>>> Am Tue, 8 Aug 2017 13:05:59 +0300 >>>>> schrieb Alexander Smirnov : >>>>> >>>>>> Add initial Isar technical overview. >>>>>> >>>>>> Signed-off-by: Alexander Smirnov >>>>>> --- >>>>>> doc/technical_overview.md | 110 >>>>>> ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, >>>>>> 110 insertions(+) create mode 100644 doc/technical_overview.md >>>>>> >>>>>> diff --git a/doc/technical_overview.md >>>>>> b/doc/technical_overview.md new file mode 100644 >>>>>> index 0000000..f65fc08 >>>>>> --- /dev/null >>>>>> +++ b/doc/technical_overview.md >>>>>> @@ -0,0 +1,110 @@ >>>>>> +# 1 Introduction >>>>>> + >>>>>> +Isar can be shortly described as a set of bitbake recipes that >>>>>> implement main build system logic. To simplify overall Isar >>>>>> understanding, this document is split into two main parts: >>>>>> + - Isar logical components >>>>>> + - Isar internal processes >>>>>> + >>>>>> +# 2 Isar Logical Components >>>>>> + >>>>>> +In this chapter the most important Isar components are >>>>>> considered in details. In this text component doesn't especially >>>>>> mean some self-contained single entity of something, it's just >>>>>> an attempt to split Isar internals by various criteria. + +### >>>>>> 2.1 Bitbake and Recipes + +All the processes in Isar are started >>>>>> by bitbake, so it manages general build executing process. >>>>>> Recipes in Isar can be split in two categories: + >>>>>> + - System recipes and classes: they are responsible for setting >>>>>> up Debian-like infrastructure, manages Debian tools execution for >>>>>> package building and installation, generation root file system >>>>>> images >>>>>> + - User recipes: custom user applications, that should be builŠµ >>>>>> from sources + >>>>>> +There are two types of dependencies in Isar: >>>>>> + - Dependency between bitbake recipes >>>>>> + - Dependencies in Debian filesystem >>>>>> + >>>>>> +**NOTE:** Isar doesn't manage dependencies in Debian file >>>>>> systems. User can only list specific Debian dependencies in >>>>>> recipe, so they eventually will be passed to apt-get or >>>>>> multistrap. Dependency installation is managed by Debian tools. >>>>>> + +### 2.2 Stamps + +Each task managed by bitbake uses stamp to >>>>>> notify that it has been completed. Due to Isar supports various >>>>>> Debian distributions and parallel builds for multiple machines >>>>>> and architectures, the stamps for tasks use special suffixes >>>>>> that include: >>>>>> + - Debian distro name >>>>>> + - Architecture >>>>>> + - Machine >>>>>> + >>>>>> +Typical example, when Isar builds the following configurations: >>>>>> + - Debian Jessie, amd64 >>>>>> + - Debian Jessie, i386 >>>>>> + - Debian Stretch, i386 >>>>>> + >>>>>> +In this case there will be 3 different buildchroots, so standard >>>>>> hello demo application should be processed 3 times for each >>>>>> environment. Three different sets of stamps should be used for >>>>>> correct bitbake operating. + +### 2.3 Buildchroot + >>>>>> +One of the key aspect of Debian philosophy claims the fact, that >>>>>> everything in Debian should be built within Debian environment. >>>>>> Moreover native compilation is more preferable than >>>>>> cross-compilation. To follow this rules, Isar introduces the new >>>>>> component - buildchroot. Bulidchroot is typical Debian filesystem >>>>>> that is created using standard Debian tools: multistrap, apt. The >>>>>> source of packages can be either official Debian repositories or >>>>>> custom repositories created by user. + +Buildchroot lifecycle can >>>>>> be described as following: >>>>>> + - Buildchroot has initial configuration file which is passed to >>>>>> multistrap tool. This configuration file is generated by bitbake >>>>>> recipe from patterns and values defined by user. Based on this >>>>>> configuration file, multistrap generates initial filesystem. >>>>>> + - During building custom Debian package, list of its build >>>>>> dependencies is installed to buildchroot. >>>>>> + - When package has been built, it's installed to current >>>>>> buildchroot to satisfy further packages build dependencies. + >>>>>> +### 2.4 Target Root Filesystem >>>>>> + >>>>>> +Target filesystem is quite similar to buildchroot. The only >>>>>> difference is that it doesn't have development packages >>>>>> installed. >>>>>> + +Target filesystem lifecycle can be described as following: >>>>>> + - Target filesystem has initial configuration file which is >>>>>> passed to multistrap tool. This configuration file is generated >>>>>> by bitbake recipe from patterns and values defined by user. >>>>>> Based on this configuration file, multistrap generates initial >>>>>> filesystem. >>>>>> + - According to the list of custom packages in bitbake recipes, >>>>>> the initial filesystem will be populated by successfully built >>>>>> packages. + +# 3 Isar Internal Processes >>>>>> + >>>>>> +### 3.1 General Overview >>>>>> + >>>>>> +Whole Isar build process can be split into the following steps: >>>>>> + - Generation of initial buildchroots for each configuration >>>>>> (Debian distro, machine and architecture) requested by user. >>>>>> + - Generation of initial target filesystems for each >>>>>> configuration. >>>>>> + - Building custom packages. >>>>>> + - Populate target filesystems. >>>>>> + - Generate bootable images. >>>>>> + >>>>>> +All these steps are described in details below. >>>>>> + >>>>>> +### 3.2 Initial Buildchroot Generation >>>>>> + >>>>>> +As mentioned above, initial buildchroot is generated using >>>>>> multistrap. The bitbake recipe which is responsible for >>>>>> buildchroot can be found here: >>>>>> `meta/recipes-devtools/buildchroot/buildchroot.bb` >>>>>> + +This recipe implementes `do_build` task which performs the >>>>>> following: +1. Generates multistrap config from template: >>>>>> `meta/recipes-devtools/buildchroot/files/multistrap.conf.in` +2. >>>>>> Install pre/post scripts for multistrap: >>>>>> `meta/recipes-devtools/buildchroot/files/configscript.sh` and >>>>>> `meta/recipes-devtools/buildchroot/files/setup.sh` +3. Run >>>>>> multistrap +4. Install script for building custom Debian >>>>>> packages: `meta/recipes-devtools/buildchroot/files/build.sh` + >>>>>> +The single stamp is created for each user buildchroot >>>>>> configuration. + +### 3.3 Initial Target Filesystem Generation + >>>>>> +Initial target filesystem generation process is very similar to >>>>>> buildchroot creating, the difference is only in initial packages >>>>>> list. + +Target image recipes are the part of Isar core. There >>>>>> is a sample of typical Isar image that can be customized >>>>>> according to the user requirements: >>>>>> `meta-isar/recipes-core/images/isar-image-base.bb` +Like for >>>>>> buildchroot, the multistrap configuration files for image can be >>>>>> found here: `meta-isar/recipes-core/images/files`, and it >>>>>> implements `do_build` task. + +### 3.4 Building Custom Packages >>>>>> + +Isar provides possibility to build Debian packages from >>>>>> sources. This features works with Debian-like source packages, >>>>>> i.e. the source code tree should contain debian folder. The >>>>>> build process is implemented in `meta/classes/dpkg.bbclass` and >>>>>> consists from the following steps: +1. Task `do_fetch`: fetch >>>>>> source code from external link +2. Task `do_unpack`: unpack >>>>>> source code to `${BUILDCHROOT_DIR}/home/build/${PN}` +3. Task >>>>>> `do_build`: switch to buildchroot using chroot command and run >>>>>> `build.sh` script. The `build.sh` script performs the following: >>>>>> + 1. Go to `/home/build/${PN}` >>>>>> + 2. Get list of dependencies from debian/control and install >>>>>> them using apt. >>>>>> + 3. Run dpkg-buildpackage >>>>>> +4. Task `do_install`: install successfully built packages >>>>>> `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy directory >>>>>> `${DEPLOY_DIR_DEB}` + +### 3.5 Populate Target Filesystem >>>>>> + >>>>>> +Each target image can be extended by custom packages listed in >>>>>> IMAGE_INSTALL variable. Task `do_populate` performs the >>>>>> following: +1. Parse IMAGE_INSTALL variable +2. Find respective >>>>>> packages in `${DEPLOY_DIR_DEB}` +3. Copy them to deb folder in >>>>>> dedicated target filesystem +4. Execute dpkg command in chroot >>>>>> for all the copied packages + >>>>>> +### 3.6 Generate Bootable Image >>>>>> + >>>>>> +This process contains the following steps: >>>>>> +1. Task `do_ext4_image`: target filesystem is packed to extfs >>>>>> image +2. wic tool generates bootable image for dedicated >>>>>> platform >>>>> >>> >> > -- With best regards, Alexander Smirnov ilbers GmbH Baierbrunner Str. 28c D-81379 Munich +49 (89) 122 67 24-0 http://ilbers.de/ Commercial register Munich, HRB 214197 General manager: Baurzhan Ismagulov