From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6451845424769662976 X-Received: by 10.46.77.7 with SMTP id a7mr532955ljb.1.1502203010597; Tue, 08 Aug 2017 07:36:50 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.25.80.87 with SMTP id z23ls185504lfj.10.gmail; Tue, 08 Aug 2017 07:36:49 -0700 (PDT) X-Received: by 10.25.92.87 with SMTP id q84mr580944lfb.43.1502203009921; Tue, 08 Aug 2017 07:36:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502203009; cv=none; d=google.com; s=arc-20160816; b=ANeBN86IiqHcG+tyueirzCOwXmEGcB97us24V4D4u0G5mPmSPpR953ljuM+DLoOLYm LOIuWlpGoItrR2ZyIyNlKajgrSPTvRTofD9o8PunS0nzKYFB51hyM/rUxgZ8E22Rolbc qAePl0prDOKFapsf4JU1VFFNREHZ7U5+1+wp7U5DQ41J4AllEuI9a1MROWHGLAS6Dg2M 3wjc42Ef3QoT0T8C5MzdXG5LfEdrllUdUVV3rykWrqH77ohQ2j+SUkUGlX+i3KTkBmTq ckzlACRJo2W5Qd5L9ZU6RJUGYXKQsJ9nG4m/sYyi7L6r7Nqwz78CeCtyxojSJt7axM8N V5AQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:arc-authentication-results; bh=x50k0ZS9p433Za/Vm1nEYJH7HFDSNO0E1uTPLOAOKlc=; b=otygmQBH7MSM8hLa/HfwsWlZcaIsJqyZqmv1KB5ePFtezctksab3kxeM9698i/sw/a 9123I+TkrASpHQbYwyGvWs6nAsclg3TsEJBnu9OyDeNuIDe2O8uv9Fddf/DBhnRa4Jg+ jrFoNcqFgO3cQkaeyYBMjEMBHPZew+I1ncOJ7CvL1mdfBt/acZVaEPMlsGt0PxT00bQs ahLzJq8RZd/GqplGfVWzo9XKHqEGjskRPmgSD8Z/kp2gjJx5pzhePH5X6xRSuedQeilr 06ItM1pBBM2JNuFAHnVCFHsuZ8TCiBQ3PFxGIdCus/D8G7zxqyKU32Cb7JJ8V6s5hD2f 6qZQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 192.35.17.28 is neither permitted nor denied by best guess record for domain of henning.schild@siemens.com) smtp.mailfrom=henning.schild@siemens.com Return-Path: Received: from goliath.siemens.de (goliath.siemens.de. [192.35.17.28]) by gmr-mx.google.com with ESMTPS id z192si539765wmz.5.2017.08.08.07.36.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 08 Aug 2017 07:36:49 -0700 (PDT) Received-SPF: neutral (google.com: 192.35.17.28 is neither permitted nor denied by best guess record for domain of henning.schild@siemens.com) client-ip=192.35.17.28; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 192.35.17.28 is neither permitted nor denied by best guess record for domain of henning.schild@siemens.com) smtp.mailfrom=henning.schild@siemens.com Received: from mail1.siemens.de (mail1.siemens.de [139.23.33.14]) by goliath.siemens.de (8.15.2/8.15.2) with ESMTPS id v78EanRp025595 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 8 Aug 2017 16:36:49 +0200 Received: from md1em3qc ([139.25.68.40]) by mail1.siemens.de (8.15.2/8.15.2) with ESMTP id v78EanUO025862; Tue, 8 Aug 2017 16:36:49 +0200 Date: Tue, 8 Aug 2017 16:38:45 +0200 From: Henning Schild To: Alexander Smirnov Cc: Subject: Re: [PATCH] doc: Add technical overview Message-ID: <20170808163845.03fc807c@md1em3qc> In-Reply-To: <287754c1-1bc6-8fb7-d83c-504adc965a08@ilbers.de> References: <20170808100559.19682-1-asmirnov@ilbers.de> <20170808131102.0ef91da0@md1em3qc> <20170808135415.6b88e860@md1em3qc> <247866c8-834d-61d6-7966-bfb702977770@ilbers.de> <20170808150313.55711c49@md1em3qc> <287754c1-1bc6-8fb7-d83c-504adc965a08@ilbers.de> X-Mailer: Claws Mail 3.13.2 (GTK+ 2.24.31; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-TUID: OQtPKwN9kE/m Am Tue, 8 Aug 2017 16:12:02 +0300 schrieb Alexander Smirnov : > On 08/08/2017 04:03 PM, Henning Schild wrote: > > Am Tue, 8 Aug 2017 15:41:10 +0300 > > schrieb Alexander Smirnov : > > =20 > >> On 08/08/2017 02:54 PM, Henning Schild wrote: =20 > >>> Am Tue, 8 Aug 2017 14:32:00 +0300 > >>> schrieb Alexander Smirnov : > >>> =20 > >>>> Hi, > >>>> > >>>> On 08/08/2017 02:11 PM, Henning Schild wrote: =20 > >>>>> I did not really read this yet. At the moment i think we have > >>>>> too many changes in the q to add such a document to the > >>>>> repository. Any change would just mean heaving to keep it up to > >>>>> date. We talked about the need for such a document for the > >>>>> possible OE integration. I think it is valuable for that > >>>>> already.=20 > >>>> > >>>> Briefly speaking, this document describes initial motivation and > >>>> high-level current overview. > >>>> =20 > >>>>> That should be added after we all are happy with the feature-set > >>>>> of Isar, until then it would just be another file to update when > >>>>> changing how things work. =20 > >>>> > >>>> Hmm, this document doesn't affect any discussions about Isar > >>>> feature set. It's intended to be as technical background for > >>>> hot-plug joining to this project. During discussions last month I > >>>> see that we have gap in our understandings how Isar works, so > >>>> this document should be start point to eliminate such gaps. > >>>> > >>>> For sure, it's *not* a specification, which explicitly defines > >>>> specific feature set and nothing more will be added. It's just an > >>>> explanation of current state. I see no problems to regularly > >>>> update it, moreover I highly appreciate this. Lack of > >>>> documentation is the most common problem of open source > >>>> projects, so as soon we start documenting the things, the easier > >>>> will be documentation maintenance in future. =20 > >>> > >>> For sure we need that! But especially section 3 contains lots of > >>> interna that will change if my patches get accepted. That is why i > >>> would like to postpone adding the file after my patches went in. =20 > >> > >> As I said, this document describes current project state, so I > >> don't think that upcoming changes should block publishing of > >> documentation. > >> > >> =20 > >>> > >>> Adding that file before them would mean another rebase round for > >>> documentation. I would be happy to rebase this one for you on top > >>> of a master that contains my stuff, but not too happy about the > >>> other way around. =20 > >> > >> The patch creates document from the scratch, so it can't conflict > >> with any upcoming patches. =20 > >=20 > > It conflicts with other patches that are under review right now. > > All i am asking is to merge conflicting patches in the order the > > patches initially showed up for review, in that case this one comes > > last anyways. And when its time has come it will be outdated, and i > > will update it for you. > > =20 >=20 > Hmm, are you speaking about exactly this patch or about previous > series? Section 3.4 and 3.5 conflict with my changes, they will have to be changed. If your patch comes first i would have to rebase. i.e. - 3.4 talks about dpkg.bbclass which i renamed [PATCH 11-16 of 16 v2 4/6] - 3.5 will not happen anymore if everything enters the rootfs through multistrap [PATCH 3/3] classes: image: remove populate and replace it with a custom repo Henning > Alex >=20 > >> Also I expect, that the author of new features will update the > >> documentation, so some reference document should be published > >> anyway. =20 > >=20 > > Sure. > > =20 > >> To be honest I'm not happy with the idea that I'll document all the > >> new features in future. =20 > >=20 > > Missing or inconsistent documentation are a valid reason to not > > accept a patch so it will not come to that. > >=20 > > Henning > > =20 > >> Alex > >> =20 > >>>>> > >>>>> Am Tue, 8 Aug 2017 13:05:59 +0300 > >>>>> schrieb Alexander Smirnov : > >>>>> =20 > >>>>>> Add initial Isar technical overview. > >>>>>> > >>>>>> Signed-off-by: Alexander Smirnov > >>>>>> --- > >>>>>> doc/technical_overview.md | 110 > >>>>>> ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, > >>>>>> 110 insertions(+) create mode 100644 doc/technical_overview.md > >>>>>> > >>>>>> diff --git a/doc/technical_overview.md > >>>>>> b/doc/technical_overview.md new file mode 100644 > >>>>>> index 0000000..f65fc08 > >>>>>> --- /dev/null > >>>>>> +++ b/doc/technical_overview.md > >>>>>> @@ -0,0 +1,110 @@ > >>>>>> +# 1 Introduction > >>>>>> + > >>>>>> +Isar can be shortly described as a set of bitbake recipes that > >>>>>> implement main build system logic. To simplify overall Isar > >>>>>> understanding, this document is split into two main parts: > >>>>>> + - Isar logical components > >>>>>> + - Isar internal processes > >>>>>> + > >>>>>> +# 2 Isar Logical Components > >>>>>> + > >>>>>> +In this chapter the most important Isar components are > >>>>>> considered in details. In this text component doesn't > >>>>>> especially mean some self-contained single entity of > >>>>>> something, it's just an attempt to split Isar internals by > >>>>>> various criteria. + +### 2.1 Bitbake and Recipes + +All the > >>>>>> processes in Isar are started by bitbake, so it manages > >>>>>> general build executing process. Recipes in Isar can be split > >>>>>> in two categories: + > >>>>>> + - System recipes and classes: they are responsible for > >>>>>> setting up Debian-like infrastructure, manages Debian tools > >>>>>> execution for package building and installation, generation > >>>>>> root file system images > >>>>>> + - User recipes: custom user applications, that should be > >>>>>> buil=D0=B5 from sources + > >>>>>> +There are two types of dependencies in Isar: > >>>>>> + - Dependency between bitbake recipes > >>>>>> + - Dependencies in Debian filesystem > >>>>>> + > >>>>>> +**NOTE:** Isar doesn't manage dependencies in Debian file > >>>>>> systems. User can only list specific Debian dependencies in > >>>>>> recipe, so they eventually will be passed to apt-get or > >>>>>> multistrap. Dependency installation is managed by Debian tools. > >>>>>> + +### 2.2 Stamps + +Each task managed by bitbake uses stamp to > >>>>>> notify that it has been completed. Due to Isar supports various > >>>>>> Debian distributions and parallel builds for multiple machines > >>>>>> and architectures, the stamps for tasks use special suffixes > >>>>>> that include: > >>>>>> + - Debian distro name > >>>>>> + - Architecture > >>>>>> + - Machine > >>>>>> + > >>>>>> +Typical example, when Isar builds the following > >>>>>> configurations: > >>>>>> + - Debian Jessie, amd64 > >>>>>> + - Debian Jessie, i386 > >>>>>> + - Debian Stretch, i386 > >>>>>> + > >>>>>> +In this case there will be 3 different buildchroots, so > >>>>>> standard hello demo application should be processed 3 times > >>>>>> for each environment. Three different sets of stamps should be > >>>>>> used for correct bitbake operating. + +### 2.3 Buildchroot + > >>>>>> +One of the key aspect of Debian philosophy claims the fact, > >>>>>> that everything in Debian should be built within Debian > >>>>>> environment. Moreover native compilation is more preferable > >>>>>> than cross-compilation. To follow this rules, Isar introduces > >>>>>> the new component - buildchroot. Bulidchroot is typical Debian > >>>>>> filesystem that is created using standard Debian tools: > >>>>>> multistrap, apt. The source of packages can be either official > >>>>>> Debian repositories or custom repositories created by user. + > >>>>>> +Buildchroot lifecycle can be described as following: > >>>>>> + - Buildchroot has initial configuration file which is passed > >>>>>> to multistrap tool. This configuration file is generated by > >>>>>> bitbake recipe from patterns and values defined by user. Based > >>>>>> on this configuration file, multistrap generates initial > >>>>>> filesystem. > >>>>>> + - During building custom Debian package, list of its build > >>>>>> dependencies is installed to buildchroot. > >>>>>> + - When package has been built, it's installed to current > >>>>>> buildchroot to satisfy further packages build dependencies. + > >>>>>> +### 2.4 Target Root Filesystem > >>>>>> + > >>>>>> +Target filesystem is quite similar to buildchroot. The only > >>>>>> difference is that it doesn't have development packages > >>>>>> installed. > >>>>>> + +Target filesystem lifecycle can be described as following: > >>>>>> + - Target filesystem has initial configuration file which is > >>>>>> passed to multistrap tool. This configuration file is generated > >>>>>> by bitbake recipe from patterns and values defined by user. > >>>>>> Based on this configuration file, multistrap generates initial > >>>>>> filesystem. > >>>>>> + - According to the list of custom packages in bitbake > >>>>>> recipes, the initial filesystem will be populated by > >>>>>> successfully built packages. + +# 3 Isar Internal Processes > >>>>>> + > >>>>>> +### 3.1 General Overview > >>>>>> + > >>>>>> +Whole Isar build process can be split into the following > >>>>>> steps: > >>>>>> + - Generation of initial buildchroots for each configuration > >>>>>> (Debian distro, machine and architecture) requested by user. > >>>>>> + - Generation of initial target filesystems for each > >>>>>> configuration. > >>>>>> + - Building custom packages. > >>>>>> + - Populate target filesystems. > >>>>>> + - Generate bootable images. > >>>>>> + > >>>>>> +All these steps are described in details below. > >>>>>> + > >>>>>> +### 3.2 Initial Buildchroot Generation > >>>>>> + > >>>>>> +As mentioned above, initial buildchroot is generated using > >>>>>> multistrap. The bitbake recipe which is responsible for > >>>>>> buildchroot can be found here: > >>>>>> `meta/recipes-devtools/buildchroot/buildchroot.bb` > >>>>>> + +This recipe implementes `do_build` task which performs the > >>>>>> following: +1. Generates multistrap config from template: > >>>>>> `meta/recipes-devtools/buildchroot/files/multistrap.conf.in` > >>>>>> +2. Install pre/post scripts for multistrap: > >>>>>> `meta/recipes-devtools/buildchroot/files/configscript.sh` and > >>>>>> `meta/recipes-devtools/buildchroot/files/setup.sh` +3. Run > >>>>>> multistrap +4. Install script for building custom Debian > >>>>>> packages: `meta/recipes-devtools/buildchroot/files/build.sh` + > >>>>>> +The single stamp is created for each user buildchroot > >>>>>> configuration. + +### 3.3 Initial Target Filesystem Generation > >>>>>> + +Initial target filesystem generation process is very > >>>>>> similar to buildchroot creating, the difference is only in > >>>>>> initial packages list. + +Target image recipes are the part of > >>>>>> Isar core. There is a sample of typical Isar image that can be > >>>>>> customized according to the user requirements: > >>>>>> `meta-isar/recipes-core/images/isar-image-base.bb` +Like for > >>>>>> buildchroot, the multistrap configuration files for image can > >>>>>> be found here: `meta-isar/recipes-core/images/files`, and it > >>>>>> implements `do_build` task. + +### 3.4 Building Custom Packages > >>>>>> + +Isar provides possibility to build Debian packages from > >>>>>> sources. This features works with Debian-like source packages, > >>>>>> i.e. the source code tree should contain debian folder. The > >>>>>> build process is implemented in `meta/classes/dpkg.bbclass` and > >>>>>> consists from the following steps: +1. Task `do_fetch`: fetch > >>>>>> source code from external link +2. Task `do_unpack`: unpack > >>>>>> source code to `${BUILDCHROOT_DIR}/home/build/${PN}` +3. Task > >>>>>> `do_build`: switch to buildchroot using chroot command and run > >>>>>> `build.sh` script. The `build.sh` script performs the > >>>>>> following: > >>>>>> + 1. Go to `/home/build/${PN}` > >>>>>> + 2. Get list of dependencies from debian/control and install > >>>>>> them using apt. > >>>>>> + 3. Run dpkg-buildpackage > >>>>>> +4. Task `do_install`: install successfully built packages > >>>>>> `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy directory > >>>>>> `${DEPLOY_DIR_DEB}` + +### 3.5 Populate Target Filesystem > >>>>>> + > >>>>>> +Each target image can be extended by custom packages listed in > >>>>>> IMAGE_INSTALL variable. Task `do_populate` performs the > >>>>>> following: +1. Parse IMAGE_INSTALL variable +2. Find respective > >>>>>> packages in `${DEPLOY_DIR_DEB}` +3. Copy them to deb folder in > >>>>>> dedicated target filesystem +4. Execute dpkg command in chroot > >>>>>> for all the copied packages + > >>>>>> +### 3.6 Generate Bootable Image > >>>>>> + > >>>>>> +This process contains the following steps: > >>>>>> +1. Task `do_ext4_image`: target filesystem is packed to extfs > >>>>>> image +2. wic tool generates bootable image for dedicated > >>>>>> platform =20 > >>>>> =20 > >>> =20 > >> =20 > > =20 >=20