From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6660477167393046528 X-Received: by 2002:a50:928a:: with SMTP id k10mr5585728eda.10.1554280388432; Wed, 03 Apr 2019 01:33:08 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a17:906:1d47:: with SMTP id o7ls192506ejh.0.gmail; Wed, 03 Apr 2019 01:33:07 -0700 (PDT) X-Google-Smtp-Source: APXvYqzCoEQoTiRN22G58tkKw0cPNJgc78ALVdVdl9ZUMnRj67EQQ1UPJB1FkzvSsQP+U4kGyrP2 X-Received: by 2002:a17:906:9613:: with SMTP id s19mr4729229ejx.13.1554280387812; Wed, 03 Apr 2019 01:33:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554280387; cv=none; d=google.com; s=arc-20160816; b=t/o5HumYYkVievjb/LVwxazqtnmr4uc3AUP5ShyZad7G+MWTKqqpf34Hmnd5FJDpJx 0vfei4wGEp8GtS5N9n/8z8LLE1l0spBETgPKnlrJ/Nm91+JPT5+1SwGmZUNn+WUjhAgt voLCuBZ4scmXm7u4YkujQ3P98Ex7frFdJlh6qq9E216bUXk+r0Cbd7b7pllR+3NnTzyh 8p5EPHihlcn1MYIORjdAjpK5cwcofGL+VqTCHO2gX6cALF86W2zwrNbJhwZGLKMxD030 NbZ4E6i9nk/aT3G8t9rI8T8memg+zedg0R3ieAQhhatBiXdMEDeepK/gUdLyp3zCj8aL yMmA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:organization:from:references:to:subject; bh=9FK2kkZlKC5vMDRUG1VulMWcst2RLjFb6c7AZdOZd6g=; b=O2AzduRQfRa0uveFXAxJJPOCXtsKlpKiAaTiJhTfoRysAtl3np1mGLEsaKDyKQQpoh zmzR93yUnHjLq5FmDwr8ZTVtbWbX2T82mcCQ5e0LHz1xzfvbPbVYm4+5JcieTBOfggI6 smFY3+e+3hIQnwPtR85Il1xq0vs8En6/QNvjc+Hlk8cF7MIuSteS3IAGEqR4aa5rj4Ij VBmP2IlDo4hjR+xQ7z3QmpYmwHoK9hVvTczad6qRZjZZ+0IVxNogn/x8QwmjznZKVqAW iu+WNTcY7M9nwMzbejQiUnX5k5dI9tKMtgVoG5C/5UgdJ9MbfdWEdW8fGNXSoMVlzHiB ivag== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of mosipov@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=mosipov@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id w13si733096edb.1.2019.04.03.01.33.07 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 03 Apr 2019 01:33:07 -0700 (PDT) Received-SPF: pass (google.com: domain of mosipov@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of mosipov@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=mosipov@ilbers.de Received: from [192.168.1.29] (4.28-242-81.adsl-dyn.isp.belgacom.be [81.242.28.4]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPSA id x338X1iW005895 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 3 Apr 2019 10:33:06 +0200 Subject: Re: [RFC] v2 Documentation: dependencies specification in Bitbake and Debian To: "Cirujano Cuesta, Silvano" , "isar-users@googlegroups.com" References: <1550763175.29605.16.camel@siemens.com> <1553865441.2782.13.camel@siemens.com> <19ce4766-3515-c0fc-e5a7-8ec7f4f56c28@ilbers.de> <1554279858.31045.46.camel@siemens.com> From: "Maxim Yu. Osipov" Organization: ilbers GmbH Message-ID: Date: Wed, 3 Apr 2019 10:32:56 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.5.1 MIME-Version: 1.0 In-Reply-To: <1554279858.31045.46.camel@siemens.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: gHoIWWsEdj/K Hi Silvano, On 4/3/19 10:24 AM, Cirujano Cuesta, Silvano wrote: > Hi Maxim, > > Thank you for the information! > I'm sorry, in the hurry I just concentrated in the content and not that much in the format of the e-mail. > I've just used wrongly my corporate e-mail client (Evolution) and > copy&paste. > But I know in general how to do it properly both with Evolution [1] and send-email and pay more attention when contributing code intended to be merged. > > I found a cover letter overkilling and therefore I intended to put my message in the part of the diff being ignored (after '---' and before the files list). > The contribution guide mentions a cover letter associated to patch series, but I still find it a bit overkilling for such a RFC. > > BTW I didn't find any information in the contribution guide about the 'scissors format' for providing some text to be ignored. > Is is accepted for single patch contributions? Sure - send it in 'scissors format' if you find it more appropriate for your patch. We are not "strict cover letter" purists :) Thanks, Maxim. > I still see the advantage of a cover letter for patch series. > > Cheers, > Silvano > > [1] https://www.kernel.org/doc/html/v5.0/process/email-clients.html?highlight=evolution#evolution-gui > > On Wed, 2019-04-03 at 09:19 +0200, Maxim Yu. Osipov wrote: >> Hi Silvano, >> >> I've tried to apply your patch but 'git am' complains about malformed line. >> >> Please have a look in the section Formatting Patches in >> https://github.com/ilbers/isar/blob/next/CONTRIBUTING.md >> >> Personally I use the following sequence of commands when submitting the >> patch (f.e. v2) to the mailing list: >> >> git format-patch --cover-letter --subject-prefix="PATCH v2" -o out -1 >> vim out/0000-cover-letter.patch >> git send-email --suppress-cc=all --to isar-users@googlegroups.com out/* >> >> In cover letter people usually mention differences versus previous >> patch series version and other stuff which shouldn't go into the commit >> message. >> >> >> Thanks, >> Maxim. >> >> >> On 3/29/19 2:17 PM, Cirujano Cuesta, Silvano wrote: >>> Reposting my first message on this thread adding the patch for mailing list review. >>> >>> >>> After my first try a couple of weeks ago, I'm coming back with a new try to improve the documentation. >>> >>> I've taken Claudius feedback into account (thank you again for your very valuable input!). >>> And I let two colleagues of mine with a similar superficial/deep knowledge of ISAR review it. >>> >>> Right now it's everything in a single MarkDown document, but I'd rather put the section "Dependency specification variables" into a separate document. >>> I keep it together right now to simplify the reviews. >>> >>> For those of you who like to have some comfort in life, I've pushed the document to a fork of ISAR in my personal area. >>> That way you can take advantage of MarkDown conversion to HTML and rendering going to [1] ;-) >>> >>> [1] https://github.com/Silvanoc/isar/blob/silvano/dependencies-documentation/doc/dependencies.md >>> >>> Signed-off-by: Silvano Cirujano Cuesta >>> --- >>>   doc/dependencies.md | 255 ++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>   1 file changed, 255 insertions(+) >>>   create mode 100644 doc/dependencies.md >>> >>> diff --git a/doc/dependencies.md b/doc/dependencies.md >>> new file mode 100644 >>> index 0000000..390e677 >>> --- /dev/null >>> +++ b/doc/dependencies.md >>> @@ -0,0 +1,255 @@ >>> +# Dependency management in ISAR >>> + >>> +## Introduction >>> + >>> +The typical goal of an ISAR build is to obtain: >>> + >>> +1. a disk image containing one or more partition images (all optional), >>> +1. containing one or more Debian root filesystems with >>> +1. certain Debian packages installed. >>> + >>> +Reaching this goal relies on the availability of the above listed elements on *runtime*, the specification of the **[runtime environments](#runtime-environment-specification)** helps the >>> different >>> build tools ensuring the availability of the mentioned elements on runtime. >>> + >>> +Usually some of the runtime dependencies have to be built, and it happens in special environments. These **[buildtime environments](#buildtime-environment-specification)** help the different >>> build >>> tools preparing the environment for building the runtime dependencies. >>> + >>> +The differentiation between buildtime and runtime environments is very important. >>> +The fact that ISAR uses Debian both for buildtime and runtime environments might mislead some people. >>> +In general the names of the variables being used to specify those dependencies aren't supporting this differentiation, but difficulting it. >>> + >>> +Not only the build environments have to be specified, but also the **[build process](#build-process-specification)** that takes care of building and integrating both the runtime dependencies and >>> the >>> buildtime dependencies. >>> + >>> +## Runtime environment specification >>> + >>> +As mentioned above, ISAR can build [partitioned disk images](#disk-image) or only [root filesystems](#root-filesystem). >>> + >>> +Let's first analyze how the content of the different runtime containers can be specified from bigger to smaller container. >>> + >>> +``` >>> ++---------------------------------------------------------------------------------------------------+ >>> +|                                                                                                   | >>> +|    DISK IMAGE                                                                                     | >>> +|                                                                                                   | >>> +|   +------------+   +-----------------------------------+   +------------------------------+       | >>> +|   |            |   |                                   |   |                              ||      | >>> +|   | Bootloader |   | Bootable partition                |   | Other partitions             |||     | >>> +|   |            |   |                                   |   |                              ||||    | >>> +|   |            |   | +------------------------------+  |   |                              ||||    | >>> +|   |            |   | |                              |  |   |                              ||||    | >>> +|   |            |   | |  ROOT FILESYSTEM             |  |   |                              ||||    | >>> +|   +------------+   | |                              |  |   |                              ||||    | >>> +|                    | |    +-------------------+     |  |   |                              ||||    | >>> +|                    | |    |                   | |   |  |   |                              ||||    | >>> +|                    | |    |  DEBIAN PACKAGES  | |   |  |   |                              ||||    | >>> +|                    | |    |                   | |   |  |   |                              ||||    | >>> +|                    | |    |                   | |   |  |   |                              ||||    | >>> +|                    | |    |                   | |   |  |   |                              ||||    | >>> +|                    | |    +-------------------+ |   |  |   |                              ||||    | >>> +|                    | |     ---------------------+   |  |   |                              ||||    | >>> +|                    | |                              |  |   +------------------------------+|||    | >>> +|                    | +------------------------------+  |    -------------------------------+||    | >>> +|                    |                                   |    --------------------------------+|    | >>> +|                    +-----------------------------------+     --------------------------------+    | >>> +|                                                                                                   | >>> ++---------------------------------------------------------------------------------------------------+ >>> +``` >>> + >>> +### Disk image >>> + >>> +A disk image depends on: >>> + >>> +1. a bootloader (optional) and >>> +1. one or more partitions, containing one of them a [root filesystem](#root-filesystem). >>> + >>> +The ISAR way to **specify** them is through *WKS files* (OpenEmbedded KickStart files). >>> +ISAR doesn't change it in any significant way, therefore the corresponding [Yocto documentation][wks] provides the best reference. >>> + >>> +### Root filesystem >>> + >>> +A root filesystem depends on: >>> + >>> +1. a minimal set of files (the result of `debootstrap`) and >>> +1. additional files and configurations obtained from *Debian packages*. >>> + >>> +The ISAR way to **specify** them is quite complex and involves *ISAR variables* that can be found in configuration files or recipes and artifacts and metadata being provided by *Debian packages*. >>> + >>> +See following sections for more information about how to specify the content of the root filesystem: >>> + >>> +* [ISAR `IMAGE_INSTALL` variable](#isar-image_install-variable) >>> +* [ISAR `IMAGE_PREINSTALL` variable](#isar-image_preinstall-variable) >>> +* [ISAR `DEBIAN_DEPENDS` variable](#isar-debian_depends-variable) >>> +* [Debian `Depends` control-file value](#debian-depends-control-file-value) >>> + >>> +## Buildtime environment specification >>> + >>> +The creation of the [above mentioned runtime units](#runtime-environment-specification) (disk images, bootloaders, root filesystems, packages, ...) takes place in different environments. >>> + >>> +These environments have to be specified. >>> + >>> +### Buildchroot >>> + >>> +The *Debian packages* are built in the so-called *buildchroot* filesystem. >>> + >>> +It's a Debian chroot that is used to build custom Debian packages required by the **[root filesystem](#root-filesystem)**. >>> +This build environment depends on the presence of certain artifacts (compilers, configurations, linkers, ...). >>> + >>> +The ISAR way to **specify** them is quite complex and involves *ISAR variables* that can be found in configuration files or recipes and artifacts and metadata being provided by *Debian packages*. >>> + >>> +See following sections for more information about how to specify the content of the chroot: >>> + >>> +* [ISAR `IMAGER_INSTALL` variable](#isar-image_install-variable) >>> +* [ISAR `BUILDCHROOT_PREINSTALL` variable](#isar-buildchroot_preinstall-variable) >>> +* [Debian `Build-Depends` control-file value](#debian-build-depends-control-file-value) >>> + >>> +If a disk image is being created, the same environment is being used for building the image. >>> +For this usage additional dependencies might appear. See the [imager](#imager) section for more information. >>> + >>> +### Imager >>> + >>> +The *disk images* are built in the so-called *imager*. >>> + >>> +It's a Debian chroot that is used to build **[disk images](#disk-image)**. >>> +This build environment depends on the presence of certain artifacts (compilers, configurations, linkers, ...). >>> +Although conceptually the imager doesn't have anything to do with the buildchroot, in the implementation the same chroot is being used for both tasks. >>> + >>> +The ISAR way to **specify** them involves *ISAR variables* that can be found in configuration files or recipes. >>> + >>> +See following sections for more information about how to specify the content of the chroot: >>> + >>> +* [ISAR `WIC_IMAGER_INSTALL` variable](#isar-wic_imager_install-variable) >>> +* [ISAR `IMAGER_BUILD_DEPS` variable](#isar-imager_build_deps-variable) >>> + >>> +### Target filesystem >>> + >>> +It's a Debian root filesystem that results from creating a minimalistic Debian skeleton via `debootstrap` and installing Debian packages on it, resulting in the [root filesystem](#root- >>> filesystem). >>> + >>> +There's no need to **specify** this build environment, since it's the [root filesystem of the runtime environment](#root-filesystem). >>> + >>> +## Build process specification >>> + >>> +ISAR relies on a subset of Bitbake to take care of the build process. >>> +Therefore the ISAR way to **specify** the different tasks involved in the process is Bitbake recipes with some *Bitbake variables* (well documented ) and some *ISAR variables* (some of them are >>> only >>> syntactic sugar). >>> + >>> +See following sections for more information about how to specify inter-tasks dependencies: >>> + >>> +* [Bitbake `DEPENDS` variable](#bitbake-depends-variable) >>> +* [ISAR `IMAGE_INSTALL` variable](#isar-image_install-variable) >>> +* [ISAR `IMAGER_BUILD_DEPS` variable](#isar-imager_build_deps-variable) >>> + >>> +## Mapping: Variables <=> Build environment and Stages >>> + >>> +| Variable | Environment | Stage | >>> +|:-------- |:-----------------:|:-----:| >>> +| [`DEPENDS`](#bitbake-depends-variable) | | All | >>> +| [`IMAGE_INSTALL`](#isar-image_install-variable) | [Root filesystem](#root-filesystem) | [Root filesystem](#root-filesystem) preparation and build | >>> +| [`IMAGE_PREINSTALL`](#isar-image_preinstall-variable) | [Root filesystem](#root-filesystem) | | >>> +| [`DEBIAN_DEPENDS`](#isar-debian_depends-variable) | [Root filesystem](#root-filesystem) | | >>> +| [`Depends` control-file](#debian-depends-control-file-value) | [Root filesystem](#root-filesystem) | | >>> +| [`IMAGE_TRANSIENT_PACKAGES`](#isar-image_transient_packages-variable) | [Root filesystem](#root-filesystem) | [Root filesystem](#root-filesystem) preparation and build | >>> +| [`BUILDCHROOT_PREINSTALL`](#isar-buildchroot_preinstall-variable) | [Buildchroot](#buildroot) | | >>> +| [`Build-Depends` control-file](#debian-build-depends-control-file-value) | [Buildchroot](#buildroot) | | >>> +| [`IMAGER_BUILD_DEPS`](#isar-imager_build_deps-variable) | | [Imager](#imager) preparation | >>> +| [`IMAGER_INSTALL`](#isar-imager_install-variable) | [Imager](#imager) | | >>> +| [`WIC_IMAGER_INSTALL`](#isar-wic_imager_install-variable) | [Imager](#imager) | | >>> + >>> +## Dependencies specification variables >>> + >>> +### Bitbake `DEPENDS` variable >>> + >>> +This is a **Bitbake** variable that specifies **Bitbake inter-task dependencies**. >>> + >>> +Since it specifies the recipes that another recipe depend upon, this variable can be specified in **package recipes** and **image recipes**. >>> + >>> +### ISAR `IMAGE_INSTALL` variable >>> + >>> +This is originally a **Bitbake** variable that has conceptually the same goal in ISAR as in Yocto. >>> +In the case of ISAR it specifies at the same time **Bitbake inter-task dependencies** and **image custom Debian package dependencies**. >>> + >>> +It basically specifies that a **[root filesystem](#root-filesystem)** (the prefix 'IMAGE' can be misleading here) requires a custom Debian package. >>> +But since custom Debian packages are being built from package recipes, this variable also specifies that an image recipe depends on package recipes. >>> + >>> +For this "magic" to work both the Debian package and the package recipe have to have the same name. >>> +Something else than this 1:1 behaviour (e.g. a recipe creating multiple Debian binary packages out of a single Debian source package) requires some hacking out of the scope of this document. >>> + >>> +This variable can be found in **configuration files** and **image recipes**. >>> + >>> +### ISAR `IMAGE_PREINSTALL` variable >>> + >>> +This is an **ISAR-only** variable that specifies **image upstream Debian package dependencies**. >>> + >>> +It specifies that a **[root filesystem](#root-filesystem)** requires certain *upstream Debian packages*, being "upstream Debian packages" ready built packages that can be downloaded from a Debian >>> package repository. >>> + >>> +This variable can be found in **configuration files** and **image recipes**. >>> + >>> +### ISAR `DEBIAN_DEPENDS` variable >>> + >>> +This is an **ISAR-only** variable that specifies **Debian inter-package dependencies** and can be used only on package recipes inheriting from the **`dpkg-raw` class**. >>> + >>> +It lists the Debian binary packages that a custom Debian package depends upon. >>> +It can be used for both upstream and custom packages. >>> +It results in an entry in the corresponding **[Debian `Depends` control-file value](#debian-depends-control-file-value)**. >>> + >>> +A common usage is for creating meta-packages that only keep dependencies on further packages synchronized. >>> + >>> +This variable can be found in **package recipes**. >>> + >>> +### ISAR `IMAGE_TRANSIENT_PACKAGES` variable >>> + >>> +This is an **ISAR-only** variable that specifies **image custom Debian package dependencies** and **Bitbake inter-task dependencies**. >>> + >>> +This is a variable that is rarely needed and only for a very specific use case! >>> +That is when a package is needed in the **[root filesystem](#root-filesystem)** during the **root filesystem build stage**, but not in the final **[root filesystem](#root-filesystem)**. >>> +Meaning that the installation is only transient, therefore its name. >>> + >>> +This variable can be found in **configuration files** and **image recipes**. >>> + >>> +### Debian `Depends` control-file value >>> + >>> +This is an **Debian** variable that specifies **Debian inter-package dependencies**. >>> + >>> +`Depends` specifies the Debian packages that are required in the [runtime environment](#runtime-environment-specification) by the package declaring the dependencies. >>> + >>> +It relies on standard Debian mechanisms, therefore the corresponding [official Debian documentation][pkg-rels] provides the best reference. >>> + >>> +### ISAR `BUILDCHROOT_PREINSTALL` variable >>> + >>> +This is an **ISAR-only** variable that specifies **buildchroot Debian package dependencies**. >>> + >>> +It specifies that **[buildchroot](#buildroot)** requires certain *Debian packages*. >>> + >>> +This variable can be found in **buildchroot recipes**. >>> + >>> +### Debian `Build-Depends` control-file value >>> + >>> +This is an **Debian** variable that specifies **Debian inter-package dependencies**. >>> + >>> +`Build-Depends` specifies the Debian packages that are required in the [buildtime environment](#buildtime-environment-specification) to build the package declaring the dependencies. >>> + >>> +It relies on standard Debian mechanisms, therefore the corresponding [official Debian documentation][pkg-rels] provides the best reference. >>> + >>> +### ISAR `IMAGER_BUILD_DEPS` variable >>> + >>> +This is an **ISAR-only** variable that specifies **Bitbake inter-task dependencies**. >>> + >>> +It has exactly the same effect as `DEPENDS`. >>> +It simply provides syntactic sugar to separate dependencies on recipes for building the runtime environment from dependencies on recipes for building the **[imager](#imager)**. >>> + >>> +This variable can be found in **configuration files**. >>> + >>> +### ISAR `IMAGER_INSTALL` variable >>> + >>> +This is an **ISAR-only** variable that specifies **imager Debian package dependencies**. >>> + >>> +It specifies that certain *Debian packages* have to be installed on the **[imager](#imager)**. >>> + >>> +This variable can be found typically in **configuration files**, although it could also be part of **image recipes**. >>> + >>> +### ISAR `WIC_IMAGER_INSTALL` variable >>> + >>> +This is an **ISAR-only** variable that specifies **imager Debian package dependencies**. >>> + >>> +It specifies that an **[imager](#imager)** requires certain *Debian packages* so that WIC can build the disk images (e.g. `parted`). >>> + >>> +This variable can be found typically in **configuration files**, although it could also be part of **image recipes**. >>> + >>> +[wks]: https://www.yoctoproject.org/docs/current/ref-manual/ref-manual.html#ref-kickstart "OpenEmbedded Kickstart (.wks) Reference" >>> +[pkg-rels]: https://www.debian.org/doc/debian-policy/ch-relationships.html "Debian policy: Declaring relationships between packages" >>> -- >>> 2.11.0 >>> >> >> >> -- >> Maxim Osipov >> ilbers GmbH >> Maria-Merian-Str. 8 >> 85521 Ottobrunn >> Germany >> +49 (151) 6517 6917 >> mosipov@ilbers.de >> http://ilbers.de/ >> Commercial register Munich, HRB 214197 >> General Manager: Baurzhan Ismagulov >> -- Maxim Osipov ilbers GmbH Maria-Merian-Str. 8 85521 Ottobrunn Germany +49 (151) 6517 6917 mosipov@ilbers.de http://ilbers.de/ Commercial register Munich, HRB 214197 General Manager: Baurzhan Ismagulov