* [RFC] v2 Documentation: dependencies specification in Bitbake and Debian
@ 2019-02-21 15:33 Cirujano Cuesta, Silvano
2019-03-28 18:30 ` Henning Schild
2019-03-29 13:17 ` Cirujano Cuesta, Silvano
0 siblings, 2 replies; 8+ messages in thread
From: Cirujano Cuesta, Silvano @ 2019-02-21 15:33 UTC (permalink / raw)
To: isar-users
Hi,
After my first try a couple of weeks ago, I'm coming back with a new try to improve the documentation.
I've taken Claudius feedback into account (thank you again for your very valuable input!).
And I let two colleagues of mine with a similar superficial/deep knowledge of ISAR review it.
Right now it's everything in a single MarkDown document, but I'd rather put the section "Dependency specification variables" into a separate document.
I keep it together right now to simplify the reviews.
For those of you who like to have some comfort in life, I've pushed the document to a fork of ISAR in my personal area.
That way you can take advantage of MarkDown conversion to HTML and rendering going to [1] ;-)
[1] https://github.com/Silvanoc/isar/blob/silvano/dependencies-documentation/doc/dependencies.md
--
With best regards,
Silvano Cirujano Cuesta
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [RFC] v2 Documentation: dependencies specification in Bitbake and Debian
2019-02-21 15:33 [RFC] v2 Documentation: dependencies specification in Bitbake and Debian Cirujano Cuesta, Silvano
@ 2019-03-28 18:30 ` Henning Schild
2019-03-29 13:02 ` Cirujano Cuesta, Silvano
2019-03-29 13:17 ` Cirujano Cuesta, Silvano
1 sibling, 1 reply; 8+ messages in thread
From: Henning Schild @ 2019-03-28 18:30 UTC (permalink / raw)
To: [ext] Cirujano Cuesta, Silvano; +Cc: isar-users
Am Thu, 21 Feb 2019 15:33:02 +0000
schrieb "[ext] Cirujano Cuesta, Silvano"
<silvano.cirujano-cuesta@siemens.com>:
> Hi,
>
> After my first try a couple of weeks ago, I'm coming back with a new
> try to improve the documentation.
>
> I've taken Claudius feedback into account (thank you again for your
> very valuable input!). And I let two colleagues of mine with a
> similar superficial/deep knowledge of ISAR review it.
>
> Right now it's everything in a single MarkDown document, but I'd
> rather put the section "Dependency specification variables" into a
> separate document. I keep it together right now to simplify the
> reviews.
>
> For those of you who like to have some comfort in life, I've pushed
> the document to a fork of ISAR in my personal area. That way you can
> take advantage of MarkDown conversion to HTML and rendering going to
> [1] ;-)
Some is ok and since the patch is not on the list, the review is not
either.
https://github.com/Silvanoc/isar/commit/b3dddefac3004c392edac14b5dd7974f411b0583
Henning
> [1]
> https://github.com/Silvanoc/isar/blob/silvano/dependencies-documentation/doc/dependencies.md
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [RFC] v2 Documentation: dependencies specification in Bitbake and Debian
2019-03-28 18:30 ` Henning Schild
@ 2019-03-29 13:02 ` Cirujano Cuesta, Silvano
0 siblings, 0 replies; 8+ messages in thread
From: Cirujano Cuesta, Silvano @ 2019-03-29 13:02 UTC (permalink / raw)
To: Schild, Henning; +Cc: isar-users
On Thu, 2019-03-28 at 19:30 +0100, Henning Schild wrote:
> Am Thu, 21 Feb 2019 15:33:02 +0000
> schrieb "[ext] Cirujano Cuesta, Silvano"
> <silvano.cirujano-cuesta@siemens.com>:
>
> > Hi,
> >
> > After my first try a couple of weeks ago, I'm coming back with a new
> > try to improve the documentation.
> >
> > I've taken Claudius feedback into account (thank you again for your
> > very valuable input!). And I let two colleagues of mine with a
> > similar superficial/deep knowledge of ISAR review it.
> >
> > Right now it's everything in a single MarkDown document, but I'd
> > rather put the section "Dependency specification variables" into a
> > separate document. I keep it together right now to simplify the
> > reviews.
> >
> > For those of you who like to have some comfort in life, I've pushed
> > the document to a fork of ISAR in my personal area. That way you can
> > take advantage of MarkDown conversion to HTML and rendering going to
> > [1] ;-)
>
> Some is ok and since the patch is not on the list, the review is not
> either.
>
> https://github.com/Silvanoc/isar/commit/b3dddefac3004c392edac14b5dd7974f411b0583
>
> Henning
Ups, sorry. Claudius already pointed it out offline, but I forgot to send the patch :-S
I don't know how to understand 'some is ok'...
I've already considered Hennings comment in Github, but I'll nevertheless send the patch in a separate message for review in the mailing list.
BR,
Silvano
>
> > [1]
> > https://github.com/Silvanoc/isar/blob/silvano/dependencies-documentation/doc/dependencies.md
> >
>
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [RFC] v2 Documentation: dependencies specification in Bitbake and Debian
2019-02-21 15:33 [RFC] v2 Documentation: dependencies specification in Bitbake and Debian Cirujano Cuesta, Silvano
2019-03-28 18:30 ` Henning Schild
@ 2019-03-29 13:17 ` Cirujano Cuesta, Silvano
2019-04-03 7:19 ` Maxim Yu. Osipov
1 sibling, 1 reply; 8+ messages in thread
From: Cirujano Cuesta, Silvano @ 2019-03-29 13:17 UTC (permalink / raw)
To: isar-users
Reposting my first message on this thread adding the patch for mailing list review.
After my first try a couple of weeks ago, I'm coming back with a new try to improve the documentation.
I've taken Claudius feedback into account (thank you again for your very valuable input!).
And I let two colleagues of mine with a similar superficial/deep knowledge of ISAR review it.
Right now it's everything in a single MarkDown document, but I'd rather put the section "Dependency specification variables" into a separate document.
I keep it together right now to simplify the reviews.
For those of you who like to have some comfort in life, I've pushed the document to a fork of ISAR in my personal area.
That way you can take advantage of MarkDown conversion to HTML and rendering going to [1] ;-)
[1] https://github.com/Silvanoc/isar/blob/silvano/dependencies-documentation/doc/dependencies.md
Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>
---
doc/dependencies.md | 255 ++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 255 insertions(+)
create mode 100644 doc/dependencies.md
diff --git a/doc/dependencies.md b/doc/dependencies.md
new file mode 100644
index 0000000..390e677
--- /dev/null
+++ b/doc/dependencies.md
@@ -0,0 +1,255 @@
+# Dependency management in ISAR
+
+## Introduction
+
+The typical goal of an ISAR build is to obtain:
+
+1. a disk image containing one or more partition images (all optional),
+1. containing one or more Debian root filesystems with
+1. certain Debian packages installed.
+
+Reaching this goal relies on the availability of the above listed elements on *runtime*, the specification of the **[runtime environments](#runtime-environment-specification)** helps the different
build tools ensuring the availability of the mentioned elements on runtime.
+
+Usually some of the runtime dependencies have to be built, and it happens in special environments. These **[buildtime environments](#buildtime-environment-specification)** help the different build
tools preparing the environment for building the runtime dependencies.
+
+The differentiation between buildtime and runtime environments is very important.
+The fact that ISAR uses Debian both for buildtime and runtime environments might mislead some people.
+In general the names of the variables being used to specify those dependencies aren't supporting this differentiation, but difficulting it.
+
+Not only the build environments have to be specified, but also the **[build process](#build-process-specification)** that takes care of building and integrating both the runtime dependencies and the
buildtime dependencies.
+
+## Runtime environment specification
+
+As mentioned above, ISAR can build [partitioned disk images](#disk-image) or only [root filesystems](#root-filesystem).
+
+Let's first analyze how the content of the different runtime containers can be specified from bigger to smaller container.
+
+```
++---------------------------------------------------------------------------------------------------+
+| |
+| DISK IMAGE |
+| |
+| +------------+ +-----------------------------------+ +------------------------------+ |
+| | | | | | || |
+| | Bootloader | | Bootable partition | | Other partitions ||| |
+| | | | | | |||| |
+| | | | +------------------------------+ | | |||| |
+| | | | | | | | |||| |
+| | | | | ROOT FILESYSTEM | | | |||| |
+| +------------+ | | | | | |||| |
+| | | +-------------------+ | | | |||| |
+| | | | | | | | | |||| |
+| | | | DEBIAN PACKAGES | | | | | |||| |
+| | | | | | | | | |||| |
+| | | | | | | | | |||| |
+| | | | | | | | | |||| |
+| | | +-------------------+ | | | | |||| |
+| | | ---------------------+ | | | |||| |
+| | | | | +------------------------------+||| |
+| | +------------------------------+ | -------------------------------+|| |
+| | | --------------------------------+| |
+| +-----------------------------------+ --------------------------------+ |
+| |
++---------------------------------------------------------------------------------------------------+
+```
+
+### Disk image
+
+A disk image depends on:
+
+1. a bootloader (optional) and
+1. one or more partitions, containing one of them a [root filesystem](#root-filesystem).
+
+The ISAR way to **specify** them is through *WKS files* (OpenEmbedded KickStart files).
+ISAR doesn't change it in any significant way, therefore the corresponding [Yocto documentation][wks] provides the best reference.
+
+### Root filesystem
+
+A root filesystem depends on:
+
+1. a minimal set of files (the result of `debootstrap`) and
+1. additional files and configurations obtained from *Debian packages*.
+
+The ISAR way to **specify** them is quite complex and involves *ISAR variables* that can be found in configuration files or recipes and artifacts and metadata being provided by *Debian packages*.
+
+See following sections for more information about how to specify the content of the root filesystem:
+
+* [ISAR `IMAGE_INSTALL` variable](#isar-image_install-variable)
+* [ISAR `IMAGE_PREINSTALL` variable](#isar-image_preinstall-variable)
+* [ISAR `DEBIAN_DEPENDS` variable](#isar-debian_depends-variable)
+* [Debian `Depends` control-file value](#debian-depends-control-file-value)
+
+## Buildtime environment specification
+
+The creation of the [above mentioned runtime units](#runtime-environment-specification) (disk images, bootloaders, root filesystems, packages, ...) takes place in different environments.
+
+These environments have to be specified.
+
+### Buildchroot
+
+The *Debian packages* are built in the so-called *buildchroot* filesystem.
+
+It's a Debian chroot that is used to build custom Debian packages required by the **[root filesystem](#root-filesystem)**.
+This build environment depends on the presence of certain artifacts (compilers, configurations, linkers, ...).
+
+The ISAR way to **specify** them is quite complex and involves *ISAR variables* that can be found in configuration files or recipes and artifacts and metadata being provided by *Debian packages*.
+
+See following sections for more information about how to specify the content of the chroot:
+
+* [ISAR `IMAGER_INSTALL` variable](#isar-image_install-variable)
+* [ISAR `BUILDCHROOT_PREINSTALL` variable](#isar-buildchroot_preinstall-variable)
+* [Debian `Build-Depends` control-file value](#debian-build-depends-control-file-value)
+
+If a disk image is being created, the same environment is being used for building the image.
+For this usage additional dependencies might appear. See the [imager](#imager) section for more information.
+
+### Imager
+
+The *disk images* are built in the so-called *imager*.
+
+It's a Debian chroot that is used to build **[disk images](#disk-image)**.
+This build environment depends on the presence of certain artifacts (compilers, configurations, linkers, ...).
+Although conceptually the imager doesn't have anything to do with the buildchroot, in the implementation the same chroot is being used for both tasks.
+
+The ISAR way to **specify** them involves *ISAR variables* that can be found in configuration files or recipes.
+
+See following sections for more information about how to specify the content of the chroot:
+
+* [ISAR `WIC_IMAGER_INSTALL` variable](#isar-wic_imager_install-variable)
+* [ISAR `IMAGER_BUILD_DEPS` variable](#isar-imager_build_deps-variable)
+
+### Target filesystem
+
+It's a Debian root filesystem that results from creating a minimalistic Debian skeleton via `debootstrap` and installing Debian packages on it, resulting in the [root filesystem](#root-filesystem).
+
+There's no need to **specify** this build environment, since it's the [root filesystem of the runtime environment](#root-filesystem).
+
+## Build process specification
+
+ISAR relies on a subset of Bitbake to take care of the build process.
+Therefore the ISAR way to **specify** the different tasks involved in the process is Bitbake recipes with some *Bitbake variables* (well documented ) and some *ISAR variables* (some of them are only
syntactic sugar).
+
+See following sections for more information about how to specify inter-tasks dependencies:
+
+* [Bitbake `DEPENDS` variable](#bitbake-depends-variable)
+* [ISAR `IMAGE_INSTALL` variable](#isar-image_install-variable)
+* [ISAR `IMAGER_BUILD_DEPS` variable](#isar-imager_build_deps-variable)
+
+## Mapping: Variables <=> Build environment and Stages
+
+| Variable | Environment | Stage |
+|:-------- |:-----------------:|:-----:|
+| [`DEPENDS`](#bitbake-depends-variable) | | All |
+| [`IMAGE_INSTALL`](#isar-image_install-variable) | [Root filesystem](#root-filesystem) | [Root filesystem](#root-filesystem) preparation and build |
+| [`IMAGE_PREINSTALL`](#isar-image_preinstall-variable) | [Root filesystem](#root-filesystem) | |
+| [`DEBIAN_DEPENDS`](#isar-debian_depends-variable) | [Root filesystem](#root-filesystem) | |
+| [`Depends` control-file](#debian-depends-control-file-value) | [Root filesystem](#root-filesystem) | |
+| [`IMAGE_TRANSIENT_PACKAGES`](#isar-image_transient_packages-variable) | [Root filesystem](#root-filesystem) | [Root filesystem](#root-filesystem) preparation and build |
+| [`BUILDCHROOT_PREINSTALL`](#isar-buildchroot_preinstall-variable) | [Buildchroot](#buildroot) | |
+| [`Build-Depends` control-file](#debian-build-depends-control-file-value) | [Buildchroot](#buildroot) | |
+| [`IMAGER_BUILD_DEPS`](#isar-imager_build_deps-variable) | | [Imager](#imager) preparation |
+| [`IMAGER_INSTALL`](#isar-imager_install-variable) | [Imager](#imager) | |
+| [`WIC_IMAGER_INSTALL`](#isar-wic_imager_install-variable) | [Imager](#imager) | |
+
+## Dependencies specification variables
+
+### Bitbake `DEPENDS` variable
+
+This is a **Bitbake** variable that specifies **Bitbake inter-task dependencies**.
+
+Since it specifies the recipes that another recipe depend upon, this variable can be specified in **package recipes** and **image recipes**.
+
+### ISAR `IMAGE_INSTALL` variable
+
+This is originally a **Bitbake** variable that has conceptually the same goal in ISAR as in Yocto.
+In the case of ISAR it specifies at the same time **Bitbake inter-task dependencies** and **image custom Debian package dependencies**.
+
+It basically specifies that a **[root filesystem](#root-filesystem)** (the prefix 'IMAGE' can be misleading here) requires a custom Debian package.
+But since custom Debian packages are being built from package recipes, this variable also specifies that an image recipe depends on package recipes.
+
+For this "magic" to work both the Debian package and the package recipe have to have the same name.
+Something else than this 1:1 behaviour (e.g. a recipe creating multiple Debian binary packages out of a single Debian source package) requires some hacking out of the scope of this document.
+
+This variable can be found in **configuration files** and **image recipes**.
+
+### ISAR `IMAGE_PREINSTALL` variable
+
+This is an **ISAR-only** variable that specifies **image upstream Debian package dependencies**.
+
+It specifies that a **[root filesystem](#root-filesystem)** requires certain *upstream Debian packages*, being "upstream Debian packages" ready built packages that can be downloaded from a Debian
package repository.
+
+This variable can be found in **configuration files** and **image recipes**.
+
+### ISAR `DEBIAN_DEPENDS` variable
+
+This is an **ISAR-only** variable that specifies **Debian inter-package dependencies** and can be used only on package recipes inheriting from the **`dpkg-raw` class**.
+
+It lists the Debian binary packages that a custom Debian package depends upon.
+It can be used for both upstream and custom packages.
+It results in an entry in the corresponding **[Debian `Depends` control-file value](#debian-depends-control-file-value)**.
+
+A common usage is for creating meta-packages that only keep dependencies on further packages synchronized.
+
+This variable can be found in **package recipes**.
+
+### ISAR `IMAGE_TRANSIENT_PACKAGES` variable
+
+This is an **ISAR-only** variable that specifies **image custom Debian package dependencies** and **Bitbake inter-task dependencies**.
+
+This is a variable that is rarely needed and only for a very specific use case!
+That is when a package is needed in the **[root filesystem](#root-filesystem)** during the **root filesystem build stage**, but not in the final **[root filesystem](#root-filesystem)**.
+Meaning that the installation is only transient, therefore its name.
+
+This variable can be found in **configuration files** and **image recipes**.
+
+### Debian `Depends` control-file value
+
+This is an **Debian** variable that specifies **Debian inter-package dependencies**.
+
+`Depends` specifies the Debian packages that are required in the [runtime environment](#runtime-environment-specification) by the package declaring the dependencies.
+
+It relies on standard Debian mechanisms, therefore the corresponding [official Debian documentation][pkg-rels] provides the best reference.
+
+### ISAR `BUILDCHROOT_PREINSTALL` variable
+
+This is an **ISAR-only** variable that specifies **buildchroot Debian package dependencies**.
+
+It specifies that **[buildchroot](#buildroot)** requires certain *Debian packages*.
+
+This variable can be found in **buildchroot recipes**.
+
+### Debian `Build-Depends` control-file value
+
+This is an **Debian** variable that specifies **Debian inter-package dependencies**.
+
+`Build-Depends` specifies the Debian packages that are required in the [buildtime environment](#buildtime-environment-specification) to build the package declaring the dependencies.
+
+It relies on standard Debian mechanisms, therefore the corresponding [official Debian documentation][pkg-rels] provides the best reference.
+
+### ISAR `IMAGER_BUILD_DEPS` variable
+
+This is an **ISAR-only** variable that specifies **Bitbake inter-task dependencies**.
+
+It has exactly the same effect as `DEPENDS`.
+It simply provides syntactic sugar to separate dependencies on recipes for building the runtime environment from dependencies on recipes for building the **[imager](#imager)**.
+
+This variable can be found in **configuration files**.
+
+### ISAR `IMAGER_INSTALL` variable
+
+This is an **ISAR-only** variable that specifies **imager Debian package dependencies**.
+
+It specifies that certain *Debian packages* have to be installed on the **[imager](#imager)**.
+
+This variable can be found typically in **configuration files**, although it could also be part of **image recipes**.
+
+### ISAR `WIC_IMAGER_INSTALL` variable
+
+This is an **ISAR-only** variable that specifies **imager Debian package dependencies**.
+
+It specifies that an **[imager](#imager)** requires certain *Debian packages* so that WIC can build the disk images (e.g. `parted`).
+
+This variable can be found typically in **configuration files**, although it could also be part of **image recipes**.
+
+[wks]: https://www.yoctoproject.org/docs/current/ref-manual/ref-manual.html#ref-kickstart "OpenEmbedded Kickstart (.wks) Reference"
+[pkg-rels]: https://www.debian.org/doc/debian-policy/ch-relationships.html "Debian policy: Declaring relationships between packages"
--
2.11.0
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [RFC] v2 Documentation: dependencies specification in Bitbake and Debian
2019-03-29 13:17 ` Cirujano Cuesta, Silvano
@ 2019-04-03 7:19 ` Maxim Yu. Osipov
2019-04-03 8:24 ` Cirujano Cuesta, Silvano
0 siblings, 1 reply; 8+ messages in thread
From: Maxim Yu. Osipov @ 2019-04-03 7:19 UTC (permalink / raw)
To: Cirujano Cuesta, Silvano, isar-users
Hi Silvano,
I've tried to apply your patch but 'git am' complains about malformed line.
Please have a look in the section Formatting Patches in
https://github.com/ilbers/isar/blob/next/CONTRIBUTING.md
Personally I use the following sequence of commands when submitting the
patch (f.e. v2) to the mailing list:
git format-patch --cover-letter --subject-prefix="PATCH v2" -o out -1
vim out/0000-cover-letter.patch
git send-email --suppress-cc=all --to isar-users@googlegroups.com out/*
In cover letter people usually mention differences versus previous
patch series version and other stuff which shouldn't go into the commit
message.
Thanks,
Maxim.
On 3/29/19 2:17 PM, Cirujano Cuesta, Silvano wrote:
> Reposting my first message on this thread adding the patch for mailing list review.
>
>
> After my first try a couple of weeks ago, I'm coming back with a new try to improve the documentation.
>
> I've taken Claudius feedback into account (thank you again for your very valuable input!).
> And I let two colleagues of mine with a similar superficial/deep knowledge of ISAR review it.
>
> Right now it's everything in a single MarkDown document, but I'd rather put the section "Dependency specification variables" into a separate document.
> I keep it together right now to simplify the reviews.
>
> For those of you who like to have some comfort in life, I've pushed the document to a fork of ISAR in my personal area.
> That way you can take advantage of MarkDown conversion to HTML and rendering going to [1] ;-)
>
> [1] https://github.com/Silvanoc/isar/blob/silvano/dependencies-documentation/doc/dependencies.md
>
> Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>
> ---
> doc/dependencies.md | 255 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 255 insertions(+)
> create mode 100644 doc/dependencies.md
>
> diff --git a/doc/dependencies.md b/doc/dependencies.md
> new file mode 100644
> index 0000000..390e677
> --- /dev/null
> +++ b/doc/dependencies.md
> @@ -0,0 +1,255 @@
> +# Dependency management in ISAR
> +
> +## Introduction
> +
> +The typical goal of an ISAR build is to obtain:
> +
> +1. a disk image containing one or more partition images (all optional),
> +1. containing one or more Debian root filesystems with
> +1. certain Debian packages installed.
> +
> +Reaching this goal relies on the availability of the above listed elements on *runtime*, the specification of the **[runtime environments](#runtime-environment-specification)** helps the different
> build tools ensuring the availability of the mentioned elements on runtime.
> +
> +Usually some of the runtime dependencies have to be built, and it happens in special environments. These **[buildtime environments](#buildtime-environment-specification)** help the different build
> tools preparing the environment for building the runtime dependencies.
> +
> +The differentiation between buildtime and runtime environments is very important.
> +The fact that ISAR uses Debian both for buildtime and runtime environments might mislead some people.
> +In general the names of the variables being used to specify those dependencies aren't supporting this differentiation, but difficulting it.
> +
> +Not only the build environments have to be specified, but also the **[build process](#build-process-specification)** that takes care of building and integrating both the runtime dependencies and the
> buildtime dependencies.
> +
> +## Runtime environment specification
> +
> +As mentioned above, ISAR can build [partitioned disk images](#disk-image) or only [root filesystems](#root-filesystem).
> +
> +Let's first analyze how the content of the different runtime containers can be specified from bigger to smaller container.
> +
> +```
> ++---------------------------------------------------------------------------------------------------+
> +| |
> +| DISK IMAGE |
> +| |
> +| +------------+ +-----------------------------------+ +------------------------------+ |
> +| | | | | | || |
> +| | Bootloader | | Bootable partition | | Other partitions ||| |
> +| | | | | | |||| |
> +| | | | +------------------------------+ | | |||| |
> +| | | | | | | | |||| |
> +| | | | | ROOT FILESYSTEM | | | |||| |
> +| +------------+ | | | | | |||| |
> +| | | +-------------------+ | | | |||| |
> +| | | | | | | | | |||| |
> +| | | | DEBIAN PACKAGES | | | | | |||| |
> +| | | | | | | | | |||| |
> +| | | | | | | | | |||| |
> +| | | | | | | | | |||| |
> +| | | +-------------------+ | | | | |||| |
> +| | | ---------------------+ | | | |||| |
> +| | | | | +------------------------------+||| |
> +| | +------------------------------+ | -------------------------------+|| |
> +| | | --------------------------------+| |
> +| +-----------------------------------+ --------------------------------+ |
> +| |
> ++---------------------------------------------------------------------------------------------------+
> +```
> +
> +### Disk image
> +
> +A disk image depends on:
> +
> +1. a bootloader (optional) and
> +1. one or more partitions, containing one of them a [root filesystem](#root-filesystem).
> +
> +The ISAR way to **specify** them is through *WKS files* (OpenEmbedded KickStart files).
> +ISAR doesn't change it in any significant way, therefore the corresponding [Yocto documentation][wks] provides the best reference.
> +
> +### Root filesystem
> +
> +A root filesystem depends on:
> +
> +1. a minimal set of files (the result of `debootstrap`) and
> +1. additional files and configurations obtained from *Debian packages*.
> +
> +The ISAR way to **specify** them is quite complex and involves *ISAR variables* that can be found in configuration files or recipes and artifacts and metadata being provided by *Debian packages*.
> +
> +See following sections for more information about how to specify the content of the root filesystem:
> +
> +* [ISAR `IMAGE_INSTALL` variable](#isar-image_install-variable)
> +* [ISAR `IMAGE_PREINSTALL` variable](#isar-image_preinstall-variable)
> +* [ISAR `DEBIAN_DEPENDS` variable](#isar-debian_depends-variable)
> +* [Debian `Depends` control-file value](#debian-depends-control-file-value)
> +
> +## Buildtime environment specification
> +
> +The creation of the [above mentioned runtime units](#runtime-environment-specification) (disk images, bootloaders, root filesystems, packages, ...) takes place in different environments.
> +
> +These environments have to be specified.
> +
> +### Buildchroot
> +
> +The *Debian packages* are built in the so-called *buildchroot* filesystem.
> +
> +It's a Debian chroot that is used to build custom Debian packages required by the **[root filesystem](#root-filesystem)**.
> +This build environment depends on the presence of certain artifacts (compilers, configurations, linkers, ...).
> +
> +The ISAR way to **specify** them is quite complex and involves *ISAR variables* that can be found in configuration files or recipes and artifacts and metadata being provided by *Debian packages*.
> +
> +See following sections for more information about how to specify the content of the chroot:
> +
> +* [ISAR `IMAGER_INSTALL` variable](#isar-image_install-variable)
> +* [ISAR `BUILDCHROOT_PREINSTALL` variable](#isar-buildchroot_preinstall-variable)
> +* [Debian `Build-Depends` control-file value](#debian-build-depends-control-file-value)
> +
> +If a disk image is being created, the same environment is being used for building the image.
> +For this usage additional dependencies might appear. See the [imager](#imager) section for more information.
> +
> +### Imager
> +
> +The *disk images* are built in the so-called *imager*.
> +
> +It's a Debian chroot that is used to build **[disk images](#disk-image)**.
> +This build environment depends on the presence of certain artifacts (compilers, configurations, linkers, ...).
> +Although conceptually the imager doesn't have anything to do with the buildchroot, in the implementation the same chroot is being used for both tasks.
> +
> +The ISAR way to **specify** them involves *ISAR variables* that can be found in configuration files or recipes.
> +
> +See following sections for more information about how to specify the content of the chroot:
> +
> +* [ISAR `WIC_IMAGER_INSTALL` variable](#isar-wic_imager_install-variable)
> +* [ISAR `IMAGER_BUILD_DEPS` variable](#isar-imager_build_deps-variable)
> +
> +### Target filesystem
> +
> +It's a Debian root filesystem that results from creating a minimalistic Debian skeleton via `debootstrap` and installing Debian packages on it, resulting in the [root filesystem](#root-filesystem).
> +
> +There's no need to **specify** this build environment, since it's the [root filesystem of the runtime environment](#root-filesystem).
> +
> +## Build process specification
> +
> +ISAR relies on a subset of Bitbake to take care of the build process.
> +Therefore the ISAR way to **specify** the different tasks involved in the process is Bitbake recipes with some *Bitbake variables* (well documented ) and some *ISAR variables* (some of them are only
> syntactic sugar).
> +
> +See following sections for more information about how to specify inter-tasks dependencies:
> +
> +* [Bitbake `DEPENDS` variable](#bitbake-depends-variable)
> +* [ISAR `IMAGE_INSTALL` variable](#isar-image_install-variable)
> +* [ISAR `IMAGER_BUILD_DEPS` variable](#isar-imager_build_deps-variable)
> +
> +## Mapping: Variables <=> Build environment and Stages
> +
> +| Variable | Environment | Stage |
> +|:-------- |:-----------------:|:-----:|
> +| [`DEPENDS`](#bitbake-depends-variable) | | All |
> +| [`IMAGE_INSTALL`](#isar-image_install-variable) | [Root filesystem](#root-filesystem) | [Root filesystem](#root-filesystem) preparation and build |
> +| [`IMAGE_PREINSTALL`](#isar-image_preinstall-variable) | [Root filesystem](#root-filesystem) | |
> +| [`DEBIAN_DEPENDS`](#isar-debian_depends-variable) | [Root filesystem](#root-filesystem) | |
> +| [`Depends` control-file](#debian-depends-control-file-value) | [Root filesystem](#root-filesystem) | |
> +| [`IMAGE_TRANSIENT_PACKAGES`](#isar-image_transient_packages-variable) | [Root filesystem](#root-filesystem) | [Root filesystem](#root-filesystem) preparation and build |
> +| [`BUILDCHROOT_PREINSTALL`](#isar-buildchroot_preinstall-variable) | [Buildchroot](#buildroot) | |
> +| [`Build-Depends` control-file](#debian-build-depends-control-file-value) | [Buildchroot](#buildroot) | |
> +| [`IMAGER_BUILD_DEPS`](#isar-imager_build_deps-variable) | | [Imager](#imager) preparation |
> +| [`IMAGER_INSTALL`](#isar-imager_install-variable) | [Imager](#imager) | |
> +| [`WIC_IMAGER_INSTALL`](#isar-wic_imager_install-variable) | [Imager](#imager) | |
> +
> +## Dependencies specification variables
> +
> +### Bitbake `DEPENDS` variable
> +
> +This is a **Bitbake** variable that specifies **Bitbake inter-task dependencies**.
> +
> +Since it specifies the recipes that another recipe depend upon, this variable can be specified in **package recipes** and **image recipes**.
> +
> +### ISAR `IMAGE_INSTALL` variable
> +
> +This is originally a **Bitbake** variable that has conceptually the same goal in ISAR as in Yocto.
> +In the case of ISAR it specifies at the same time **Bitbake inter-task dependencies** and **image custom Debian package dependencies**.
> +
> +It basically specifies that a **[root filesystem](#root-filesystem)** (the prefix 'IMAGE' can be misleading here) requires a custom Debian package.
> +But since custom Debian packages are being built from package recipes, this variable also specifies that an image recipe depends on package recipes.
> +
> +For this "magic" to work both the Debian package and the package recipe have to have the same name.
> +Something else than this 1:1 behaviour (e.g. a recipe creating multiple Debian binary packages out of a single Debian source package) requires some hacking out of the scope of this document.
> +
> +This variable can be found in **configuration files** and **image recipes**.
> +
> +### ISAR `IMAGE_PREINSTALL` variable
> +
> +This is an **ISAR-only** variable that specifies **image upstream Debian package dependencies**.
> +
> +It specifies that a **[root filesystem](#root-filesystem)** requires certain *upstream Debian packages*, being "upstream Debian packages" ready built packages that can be downloaded from a Debian
> package repository.
> +
> +This variable can be found in **configuration files** and **image recipes**.
> +
> +### ISAR `DEBIAN_DEPENDS` variable
> +
> +This is an **ISAR-only** variable that specifies **Debian inter-package dependencies** and can be used only on package recipes inheriting from the **`dpkg-raw` class**.
> +
> +It lists the Debian binary packages that a custom Debian package depends upon.
> +It can be used for both upstream and custom packages.
> +It results in an entry in the corresponding **[Debian `Depends` control-file value](#debian-depends-control-file-value)**.
> +
> +A common usage is for creating meta-packages that only keep dependencies on further packages synchronized.
> +
> +This variable can be found in **package recipes**.
> +
> +### ISAR `IMAGE_TRANSIENT_PACKAGES` variable
> +
> +This is an **ISAR-only** variable that specifies **image custom Debian package dependencies** and **Bitbake inter-task dependencies**.
> +
> +This is a variable that is rarely needed and only for a very specific use case!
> +That is when a package is needed in the **[root filesystem](#root-filesystem)** during the **root filesystem build stage**, but not in the final **[root filesystem](#root-filesystem)**.
> +Meaning that the installation is only transient, therefore its name.
> +
> +This variable can be found in **configuration files** and **image recipes**.
> +
> +### Debian `Depends` control-file value
> +
> +This is an **Debian** variable that specifies **Debian inter-package dependencies**.
> +
> +`Depends` specifies the Debian packages that are required in the [runtime environment](#runtime-environment-specification) by the package declaring the dependencies.
> +
> +It relies on standard Debian mechanisms, therefore the corresponding [official Debian documentation][pkg-rels] provides the best reference.
> +
> +### ISAR `BUILDCHROOT_PREINSTALL` variable
> +
> +This is an **ISAR-only** variable that specifies **buildchroot Debian package dependencies**.
> +
> +It specifies that **[buildchroot](#buildroot)** requires certain *Debian packages*.
> +
> +This variable can be found in **buildchroot recipes**.
> +
> +### Debian `Build-Depends` control-file value
> +
> +This is an **Debian** variable that specifies **Debian inter-package dependencies**.
> +
> +`Build-Depends` specifies the Debian packages that are required in the [buildtime environment](#buildtime-environment-specification) to build the package declaring the dependencies.
> +
> +It relies on standard Debian mechanisms, therefore the corresponding [official Debian documentation][pkg-rels] provides the best reference.
> +
> +### ISAR `IMAGER_BUILD_DEPS` variable
> +
> +This is an **ISAR-only** variable that specifies **Bitbake inter-task dependencies**.
> +
> +It has exactly the same effect as `DEPENDS`.
> +It simply provides syntactic sugar to separate dependencies on recipes for building the runtime environment from dependencies on recipes for building the **[imager](#imager)**.
> +
> +This variable can be found in **configuration files**.
> +
> +### ISAR `IMAGER_INSTALL` variable
> +
> +This is an **ISAR-only** variable that specifies **imager Debian package dependencies**.
> +
> +It specifies that certain *Debian packages* have to be installed on the **[imager](#imager)**.
> +
> +This variable can be found typically in **configuration files**, although it could also be part of **image recipes**.
> +
> +### ISAR `WIC_IMAGER_INSTALL` variable
> +
> +This is an **ISAR-only** variable that specifies **imager Debian package dependencies**.
> +
> +It specifies that an **[imager](#imager)** requires certain *Debian packages* so that WIC can build the disk images (e.g. `parted`).
> +
> +This variable can be found typically in **configuration files**, although it could also be part of **image recipes**.
> +
> +[wks]: https://www.yoctoproject.org/docs/current/ref-manual/ref-manual.html#ref-kickstart "OpenEmbedded Kickstart (.wks) Reference"
> +[pkg-rels]: https://www.debian.org/doc/debian-policy/ch-relationships.html "Debian policy: Declaring relationships between packages"
> --
> 2.11.0
>
--
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [RFC] v2 Documentation: dependencies specification in Bitbake and Debian
2019-04-03 7:19 ` Maxim Yu. Osipov
@ 2019-04-03 8:24 ` Cirujano Cuesta, Silvano
2019-04-03 8:32 ` Maxim Yu. Osipov
0 siblings, 1 reply; 8+ messages in thread
From: Cirujano Cuesta, Silvano @ 2019-04-03 8:24 UTC (permalink / raw)
To: mosipov, isar-users
Hi Maxim,
Thank you for the information!
I'm sorry, in the hurry I just concentrated in the content and not that much in the format of the e-mail.
I've just used wrongly my corporate e-mail client (Evolution) and
copy&paste.
But I know in general how to do it properly both with Evolution [1] and send-email and pay more attention when contributing code intended to be merged.
I found a cover letter overkilling and therefore I intended to put my message in the part of the diff being ignored (after '---' and before the files list).
The contribution guide mentions a cover letter associated to patch series, but I still find it a bit overkilling for such a RFC.
BTW I didn't find any information in the contribution guide about the 'scissors format' for providing some text to be ignored.
Is is accepted for single patch contributions?
I still see the advantage of a cover letter for patch series.
Cheers,
Silvano
[1] https://www.kernel.org/doc/html/v5.0/process/email-clients.html?highlight=evolution#evolution-gui
On Wed, 2019-04-03 at 09:19 +0200, Maxim Yu. Osipov wrote:
> Hi Silvano,
>
> I've tried to apply your patch but 'git am' complains about malformed line.
>
> Please have a look in the section Formatting Patches in
> https://github.com/ilbers/isar/blob/next/CONTRIBUTING.md
>
> Personally I use the following sequence of commands when submitting the
> patch (f.e. v2) to the mailing list:
>
> git format-patch --cover-letter --subject-prefix="PATCH v2" -o out -1
> vim out/0000-cover-letter.patch
> git send-email --suppress-cc=all --to isar-users@googlegroups.com out/*
>
> In cover letter people usually mention differences versus previous
> patch series version and other stuff which shouldn't go into the commit
> message.
>
>
> Thanks,
> Maxim.
>
>
> On 3/29/19 2:17 PM, Cirujano Cuesta, Silvano wrote:
> > Reposting my first message on this thread adding the patch for mailing list review.
> >
> >
> > After my first try a couple of weeks ago, I'm coming back with a new try to improve the documentation.
> >
> > I've taken Claudius feedback into account (thank you again for your very valuable input!).
> > And I let two colleagues of mine with a similar superficial/deep knowledge of ISAR review it.
> >
> > Right now it's everything in a single MarkDown document, but I'd rather put the section "Dependency specification variables" into a separate document.
> > I keep it together right now to simplify the reviews.
> >
> > For those of you who like to have some comfort in life, I've pushed the document to a fork of ISAR in my personal area.
> > That way you can take advantage of MarkDown conversion to HTML and rendering going to [1] ;-)
> >
> > [1] https://github.com/Silvanoc/isar/blob/silvano/dependencies-documentation/doc/dependencies.md
> >
> > Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>
> > ---
> > doc/dependencies.md | 255 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> > 1 file changed, 255 insertions(+)
> > create mode 100644 doc/dependencies.md
> >
> > diff --git a/doc/dependencies.md b/doc/dependencies.md
> > new file mode 100644
> > index 0000000..390e677
> > --- /dev/null
> > +++ b/doc/dependencies.md
> > @@ -0,0 +1,255 @@
> > +# Dependency management in ISAR
> > +
> > +## Introduction
> > +
> > +The typical goal of an ISAR build is to obtain:
> > +
> > +1. a disk image containing one or more partition images (all optional),
> > +1. containing one or more Debian root filesystems with
> > +1. certain Debian packages installed.
> > +
> > +Reaching this goal relies on the availability of the above listed elements on *runtime*, the specification of the **[runtime environments](#runtime-environment-specification)** helps the
> > different
> > build tools ensuring the availability of the mentioned elements on runtime.
> > +
> > +Usually some of the runtime dependencies have to be built, and it happens in special environments. These **[buildtime environments](#buildtime-environment-specification)** help the different
> > build
> > tools preparing the environment for building the runtime dependencies.
> > +
> > +The differentiation between buildtime and runtime environments is very important.
> > +The fact that ISAR uses Debian both for buildtime and runtime environments might mislead some people.
> > +In general the names of the variables being used to specify those dependencies aren't supporting this differentiation, but difficulting it.
> > +
> > +Not only the build environments have to be specified, but also the **[build process](#build-process-specification)** that takes care of building and integrating both the runtime dependencies and
> > the
> > buildtime dependencies.
> > +
> > +## Runtime environment specification
> > +
> > +As mentioned above, ISAR can build [partitioned disk images](#disk-image) or only [root filesystems](#root-filesystem).
> > +
> > +Let's first analyze how the content of the different runtime containers can be specified from bigger to smaller container.
> > +
> > +```
> > ++---------------------------------------------------------------------------------------------------+
> > +| |
> > +| DISK IMAGE |
> > +| |
> > +| +------------+ +-----------------------------------+ +------------------------------+ |
> > +| | | | | | || |
> > +| | Bootloader | | Bootable partition | | Other partitions ||| |
> > +| | | | | | |||| |
> > +| | | | +------------------------------+ | | |||| |
> > +| | | | | | | | |||| |
> > +| | | | | ROOT FILESYSTEM | | | |||| |
> > +| +------------+ | | | | | |||| |
> > +| | | +-------------------+ | | | |||| |
> > +| | | | | | | | | |||| |
> > +| | | | DEBIAN PACKAGES | | | | | |||| |
> > +| | | | | | | | | |||| |
> > +| | | | | | | | | |||| |
> > +| | | | | | | | | |||| |
> > +| | | +-------------------+ | | | | |||| |
> > +| | | ---------------------+ | | | |||| |
> > +| | | | | +------------------------------+||| |
> > +| | +------------------------------+ | -------------------------------+|| |
> > +| | | --------------------------------+| |
> > +| +-----------------------------------+ --------------------------------+ |
> > +| |
> > ++---------------------------------------------------------------------------------------------------+
> > +```
> > +
> > +### Disk image
> > +
> > +A disk image depends on:
> > +
> > +1. a bootloader (optional) and
> > +1. one or more partitions, containing one of them a [root filesystem](#root-filesystem).
> > +
> > +The ISAR way to **specify** them is through *WKS files* (OpenEmbedded KickStart files).
> > +ISAR doesn't change it in any significant way, therefore the corresponding [Yocto documentation][wks] provides the best reference.
> > +
> > +### Root filesystem
> > +
> > +A root filesystem depends on:
> > +
> > +1. a minimal set of files (the result of `debootstrap`) and
> > +1. additional files and configurations obtained from *Debian packages*.
> > +
> > +The ISAR way to **specify** them is quite complex and involves *ISAR variables* that can be found in configuration files or recipes and artifacts and metadata being provided by *Debian packages*.
> > +
> > +See following sections for more information about how to specify the content of the root filesystem:
> > +
> > +* [ISAR `IMAGE_INSTALL` variable](#isar-image_install-variable)
> > +* [ISAR `IMAGE_PREINSTALL` variable](#isar-image_preinstall-variable)
> > +* [ISAR `DEBIAN_DEPENDS` variable](#isar-debian_depends-variable)
> > +* [Debian `Depends` control-file value](#debian-depends-control-file-value)
> > +
> > +## Buildtime environment specification
> > +
> > +The creation of the [above mentioned runtime units](#runtime-environment-specification) (disk images, bootloaders, root filesystems, packages, ...) takes place in different environments.
> > +
> > +These environments have to be specified.
> > +
> > +### Buildchroot
> > +
> > +The *Debian packages* are built in the so-called *buildchroot* filesystem.
> > +
> > +It's a Debian chroot that is used to build custom Debian packages required by the **[root filesystem](#root-filesystem)**.
> > +This build environment depends on the presence of certain artifacts (compilers, configurations, linkers, ...).
> > +
> > +The ISAR way to **specify** them is quite complex and involves *ISAR variables* that can be found in configuration files or recipes and artifacts and metadata being provided by *Debian packages*.
> > +
> > +See following sections for more information about how to specify the content of the chroot:
> > +
> > +* [ISAR `IMAGER_INSTALL` variable](#isar-image_install-variable)
> > +* [ISAR `BUILDCHROOT_PREINSTALL` variable](#isar-buildchroot_preinstall-variable)
> > +* [Debian `Build-Depends` control-file value](#debian-build-depends-control-file-value)
> > +
> > +If a disk image is being created, the same environment is being used for building the image.
> > +For this usage additional dependencies might appear. See the [imager](#imager) section for more information.
> > +
> > +### Imager
> > +
> > +The *disk images* are built in the so-called *imager*.
> > +
> > +It's a Debian chroot that is used to build **[disk images](#disk-image)**.
> > +This build environment depends on the presence of certain artifacts (compilers, configurations, linkers, ...).
> > +Although conceptually the imager doesn't have anything to do with the buildchroot, in the implementation the same chroot is being used for both tasks.
> > +
> > +The ISAR way to **specify** them involves *ISAR variables* that can be found in configuration files or recipes.
> > +
> > +See following sections for more information about how to specify the content of the chroot:
> > +
> > +* [ISAR `WIC_IMAGER_INSTALL` variable](#isar-wic_imager_install-variable)
> > +* [ISAR `IMAGER_BUILD_DEPS` variable](#isar-imager_build_deps-variable)
> > +
> > +### Target filesystem
> > +
> > +It's a Debian root filesystem that results from creating a minimalistic Debian skeleton via `debootstrap` and installing Debian packages on it, resulting in the [root filesystem](#root-
> > filesystem).
> > +
> > +There's no need to **specify** this build environment, since it's the [root filesystem of the runtime environment](#root-filesystem).
> > +
> > +## Build process specification
> > +
> > +ISAR relies on a subset of Bitbake to take care of the build process.
> > +Therefore the ISAR way to **specify** the different tasks involved in the process is Bitbake recipes with some *Bitbake variables* (well documented ) and some *ISAR variables* (some of them are
> > only
> > syntactic sugar).
> > +
> > +See following sections for more information about how to specify inter-tasks dependencies:
> > +
> > +* [Bitbake `DEPENDS` variable](#bitbake-depends-variable)
> > +* [ISAR `IMAGE_INSTALL` variable](#isar-image_install-variable)
> > +* [ISAR `IMAGER_BUILD_DEPS` variable](#isar-imager_build_deps-variable)
> > +
> > +## Mapping: Variables <=> Build environment and Stages
> > +
> > +| Variable | Environment | Stage |
> > +|:-------- |:-----------------:|:-----:|
> > +| [`DEPENDS`](#bitbake-depends-variable) | | All |
> > +| [`IMAGE_INSTALL`](#isar-image_install-variable) | [Root filesystem](#root-filesystem) | [Root filesystem](#root-filesystem) preparation and build |
> > +| [`IMAGE_PREINSTALL`](#isar-image_preinstall-variable) | [Root filesystem](#root-filesystem) | |
> > +| [`DEBIAN_DEPENDS`](#isar-debian_depends-variable) | [Root filesystem](#root-filesystem) | |
> > +| [`Depends` control-file](#debian-depends-control-file-value) | [Root filesystem](#root-filesystem) | |
> > +| [`IMAGE_TRANSIENT_PACKAGES`](#isar-image_transient_packages-variable) | [Root filesystem](#root-filesystem) | [Root filesystem](#root-filesystem) preparation and build |
> > +| [`BUILDCHROOT_PREINSTALL`](#isar-buildchroot_preinstall-variable) | [Buildchroot](#buildroot) | |
> > +| [`Build-Depends` control-file](#debian-build-depends-control-file-value) | [Buildchroot](#buildroot) | |
> > +| [`IMAGER_BUILD_DEPS`](#isar-imager_build_deps-variable) | | [Imager](#imager) preparation |
> > +| [`IMAGER_INSTALL`](#isar-imager_install-variable) | [Imager](#imager) | |
> > +| [`WIC_IMAGER_INSTALL`](#isar-wic_imager_install-variable) | [Imager](#imager) | |
> > +
> > +## Dependencies specification variables
> > +
> > +### Bitbake `DEPENDS` variable
> > +
> > +This is a **Bitbake** variable that specifies **Bitbake inter-task dependencies**.
> > +
> > +Since it specifies the recipes that another recipe depend upon, this variable can be specified in **package recipes** and **image recipes**.
> > +
> > +### ISAR `IMAGE_INSTALL` variable
> > +
> > +This is originally a **Bitbake** variable that has conceptually the same goal in ISAR as in Yocto.
> > +In the case of ISAR it specifies at the same time **Bitbake inter-task dependencies** and **image custom Debian package dependencies**.
> > +
> > +It basically specifies that a **[root filesystem](#root-filesystem)** (the prefix 'IMAGE' can be misleading here) requires a custom Debian package.
> > +But since custom Debian packages are being built from package recipes, this variable also specifies that an image recipe depends on package recipes.
> > +
> > +For this "magic" to work both the Debian package and the package recipe have to have the same name.
> > +Something else than this 1:1 behaviour (e.g. a recipe creating multiple Debian binary packages out of a single Debian source package) requires some hacking out of the scope of this document.
> > +
> > +This variable can be found in **configuration files** and **image recipes**.
> > +
> > +### ISAR `IMAGE_PREINSTALL` variable
> > +
> > +This is an **ISAR-only** variable that specifies **image upstream Debian package dependencies**.
> > +
> > +It specifies that a **[root filesystem](#root-filesystem)** requires certain *upstream Debian packages*, being "upstream Debian packages" ready built packages that can be downloaded from a Debian
> > package repository.
> > +
> > +This variable can be found in **configuration files** and **image recipes**.
> > +
> > +### ISAR `DEBIAN_DEPENDS` variable
> > +
> > +This is an **ISAR-only** variable that specifies **Debian inter-package dependencies** and can be used only on package recipes inheriting from the **`dpkg-raw` class**.
> > +
> > +It lists the Debian binary packages that a custom Debian package depends upon.
> > +It can be used for both upstream and custom packages.
> > +It results in an entry in the corresponding **[Debian `Depends` control-file value](#debian-depends-control-file-value)**.
> > +
> > +A common usage is for creating meta-packages that only keep dependencies on further packages synchronized.
> > +
> > +This variable can be found in **package recipes**.
> > +
> > +### ISAR `IMAGE_TRANSIENT_PACKAGES` variable
> > +
> > +This is an **ISAR-only** variable that specifies **image custom Debian package dependencies** and **Bitbake inter-task dependencies**.
> > +
> > +This is a variable that is rarely needed and only for a very specific use case!
> > +That is when a package is needed in the **[root filesystem](#root-filesystem)** during the **root filesystem build stage**, but not in the final **[root filesystem](#root-filesystem)**.
> > +Meaning that the installation is only transient, therefore its name.
> > +
> > +This variable can be found in **configuration files** and **image recipes**.
> > +
> > +### Debian `Depends` control-file value
> > +
> > +This is an **Debian** variable that specifies **Debian inter-package dependencies**.
> > +
> > +`Depends` specifies the Debian packages that are required in the [runtime environment](#runtime-environment-specification) by the package declaring the dependencies.
> > +
> > +It relies on standard Debian mechanisms, therefore the corresponding [official Debian documentation][pkg-rels] provides the best reference.
> > +
> > +### ISAR `BUILDCHROOT_PREINSTALL` variable
> > +
> > +This is an **ISAR-only** variable that specifies **buildchroot Debian package dependencies**.
> > +
> > +It specifies that **[buildchroot](#buildroot)** requires certain *Debian packages*.
> > +
> > +This variable can be found in **buildchroot recipes**.
> > +
> > +### Debian `Build-Depends` control-file value
> > +
> > +This is an **Debian** variable that specifies **Debian inter-package dependencies**.
> > +
> > +`Build-Depends` specifies the Debian packages that are required in the [buildtime environment](#buildtime-environment-specification) to build the package declaring the dependencies.
> > +
> > +It relies on standard Debian mechanisms, therefore the corresponding [official Debian documentation][pkg-rels] provides the best reference.
> > +
> > +### ISAR `IMAGER_BUILD_DEPS` variable
> > +
> > +This is an **ISAR-only** variable that specifies **Bitbake inter-task dependencies**.
> > +
> > +It has exactly the same effect as `DEPENDS`.
> > +It simply provides syntactic sugar to separate dependencies on recipes for building the runtime environment from dependencies on recipes for building the **[imager](#imager)**.
> > +
> > +This variable can be found in **configuration files**.
> > +
> > +### ISAR `IMAGER_INSTALL` variable
> > +
> > +This is an **ISAR-only** variable that specifies **imager Debian package dependencies**.
> > +
> > +It specifies that certain *Debian packages* have to be installed on the **[imager](#imager)**.
> > +
> > +This variable can be found typically in **configuration files**, although it could also be part of **image recipes**.
> > +
> > +### ISAR `WIC_IMAGER_INSTALL` variable
> > +
> > +This is an **ISAR-only** variable that specifies **imager Debian package dependencies**.
> > +
> > +It specifies that an **[imager](#imager)** requires certain *Debian packages* so that WIC can build the disk images (e.g. `parted`).
> > +
> > +This variable can be found typically in **configuration files**, although it could also be part of **image recipes**.
> > +
> > +[wks]: https://www.yoctoproject.org/docs/current/ref-manual/ref-manual.html#ref-kickstart "OpenEmbedded Kickstart (.wks) Reference"
> > +[pkg-rels]: https://www.debian.org/doc/debian-policy/ch-relationships.html "Debian policy: Declaring relationships between packages"
> > --
> > 2.11.0
> >
>
>
> --
> Maxim Osipov
> ilbers GmbH
> Maria-Merian-Str. 8
> 85521 Ottobrunn
> Germany
> +49 (151) 6517 6917
> mosipov@ilbers.de
> http://ilbers.de/
> Commercial register Munich, HRB 214197
> General Manager: Baurzhan Ismagulov
>
--
With best regards,
Silvano Cirujano Cuesta
Siemens AG
Smart Embedded Systems
Mobile: +49 173 3181462
mailto:silvano.cirujano-cuesta@siemens.com
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [RFC] v2 Documentation: dependencies specification in Bitbake and Debian
2019-04-03 8:24 ` Cirujano Cuesta, Silvano
@ 2019-04-03 8:32 ` Maxim Yu. Osipov
2019-04-03 9:06 ` Cirujano Cuesta, Silvano
0 siblings, 1 reply; 8+ messages in thread
From: Maxim Yu. Osipov @ 2019-04-03 8:32 UTC (permalink / raw)
To: Cirujano Cuesta, Silvano, isar-users
Hi Silvano,
On 4/3/19 10:24 AM, Cirujano Cuesta, Silvano wrote:
> Hi Maxim,
>
> Thank you for the information!
> I'm sorry, in the hurry I just concentrated in the content and not that much in the format of the e-mail.
> I've just used wrongly my corporate e-mail client (Evolution) and
> copy&paste.
> But I know in general how to do it properly both with Evolution [1] and send-email and pay more attention when contributing code intended to be merged.
>
> I found a cover letter overkilling and therefore I intended to put my message in the part of the diff being ignored (after '---' and before the files list).
> The contribution guide mentions a cover letter associated to patch series, but I still find it a bit overkilling for such a RFC.
>
> BTW I didn't find any information in the contribution guide about the 'scissors format' for providing some text to be ignored.
> Is is accepted for single patch contributions?
Sure - send it in 'scissors format' if you find it more appropriate for
your patch. We are not "strict cover letter" purists :)
Thanks,
Maxim.
> I still see the advantage of a cover letter for patch series.
>
> Cheers,
> Silvano
>
> [1] https://www.kernel.org/doc/html/v5.0/process/email-clients.html?highlight=evolution#evolution-gui
>
> On Wed, 2019-04-03 at 09:19 +0200, Maxim Yu. Osipov wrote:
>> Hi Silvano,
>>
>> I've tried to apply your patch but 'git am' complains about malformed line.
>>
>> Please have a look in the section Formatting Patches in
>> https://github.com/ilbers/isar/blob/next/CONTRIBUTING.md
>>
>> Personally I use the following sequence of commands when submitting the
>> patch (f.e. v2) to the mailing list:
>>
>> git format-patch --cover-letter --subject-prefix="PATCH v2" -o out -1
>> vim out/0000-cover-letter.patch
>> git send-email --suppress-cc=all --to isar-users@googlegroups.com out/*
>>
>> In cover letter people usually mention differences versus previous
>> patch series version and other stuff which shouldn't go into the commit
>> message.
>>
>>
>> Thanks,
>> Maxim.
>>
>>
>> On 3/29/19 2:17 PM, Cirujano Cuesta, Silvano wrote:
>>> Reposting my first message on this thread adding the patch for mailing list review.
>>>
>>>
>>> After my first try a couple of weeks ago, I'm coming back with a new try to improve the documentation.
>>>
>>> I've taken Claudius feedback into account (thank you again for your very valuable input!).
>>> And I let two colleagues of mine with a similar superficial/deep knowledge of ISAR review it.
>>>
>>> Right now it's everything in a single MarkDown document, but I'd rather put the section "Dependency specification variables" into a separate document.
>>> I keep it together right now to simplify the reviews.
>>>
>>> For those of you who like to have some comfort in life, I've pushed the document to a fork of ISAR in my personal area.
>>> That way you can take advantage of MarkDown conversion to HTML and rendering going to [1] ;-)
>>>
>>> [1] https://github.com/Silvanoc/isar/blob/silvano/dependencies-documentation/doc/dependencies.md
>>>
>>> Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>
>>> ---
>>> doc/dependencies.md | 255 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> 1 file changed, 255 insertions(+)
>>> create mode 100644 doc/dependencies.md
>>>
>>> diff --git a/doc/dependencies.md b/doc/dependencies.md
>>> new file mode 100644
>>> index 0000000..390e677
>>> --- /dev/null
>>> +++ b/doc/dependencies.md
>>> @@ -0,0 +1,255 @@
>>> +# Dependency management in ISAR
>>> +
>>> +## Introduction
>>> +
>>> +The typical goal of an ISAR build is to obtain:
>>> +
>>> +1. a disk image containing one or more partition images (all optional),
>>> +1. containing one or more Debian root filesystems with
>>> +1. certain Debian packages installed.
>>> +
>>> +Reaching this goal relies on the availability of the above listed elements on *runtime*, the specification of the **[runtime environments](#runtime-environment-specification)** helps the
>>> different
>>> build tools ensuring the availability of the mentioned elements on runtime.
>>> +
>>> +Usually some of the runtime dependencies have to be built, and it happens in special environments. These **[buildtime environments](#buildtime-environment-specification)** help the different
>>> build
>>> tools preparing the environment for building the runtime dependencies.
>>> +
>>> +The differentiation between buildtime and runtime environments is very important.
>>> +The fact that ISAR uses Debian both for buildtime and runtime environments might mislead some people.
>>> +In general the names of the variables being used to specify those dependencies aren't supporting this differentiation, but difficulting it.
>>> +
>>> +Not only the build environments have to be specified, but also the **[build process](#build-process-specification)** that takes care of building and integrating both the runtime dependencies and
>>> the
>>> buildtime dependencies.
>>> +
>>> +## Runtime environment specification
>>> +
>>> +As mentioned above, ISAR can build [partitioned disk images](#disk-image) or only [root filesystems](#root-filesystem).
>>> +
>>> +Let's first analyze how the content of the different runtime containers can be specified from bigger to smaller container.
>>> +
>>> +```
>>> ++---------------------------------------------------------------------------------------------------+
>>> +| |
>>> +| DISK IMAGE |
>>> +| |
>>> +| +------------+ +-----------------------------------+ +------------------------------+ |
>>> +| | | | | | || |
>>> +| | Bootloader | | Bootable partition | | Other partitions ||| |
>>> +| | | | | | |||| |
>>> +| | | | +------------------------------+ | | |||| |
>>> +| | | | | | | | |||| |
>>> +| | | | | ROOT FILESYSTEM | | | |||| |
>>> +| +------------+ | | | | | |||| |
>>> +| | | +-------------------+ | | | |||| |
>>> +| | | | | | | | | |||| |
>>> +| | | | DEBIAN PACKAGES | | | | | |||| |
>>> +| | | | | | | | | |||| |
>>> +| | | | | | | | | |||| |
>>> +| | | | | | | | | |||| |
>>> +| | | +-------------------+ | | | | |||| |
>>> +| | | ---------------------+ | | | |||| |
>>> +| | | | | +------------------------------+||| |
>>> +| | +------------------------------+ | -------------------------------+|| |
>>> +| | | --------------------------------+| |
>>> +| +-----------------------------------+ --------------------------------+ |
>>> +| |
>>> ++---------------------------------------------------------------------------------------------------+
>>> +```
>>> +
>>> +### Disk image
>>> +
>>> +A disk image depends on:
>>> +
>>> +1. a bootloader (optional) and
>>> +1. one or more partitions, containing one of them a [root filesystem](#root-filesystem).
>>> +
>>> +The ISAR way to **specify** them is through *WKS files* (OpenEmbedded KickStart files).
>>> +ISAR doesn't change it in any significant way, therefore the corresponding [Yocto documentation][wks] provides the best reference.
>>> +
>>> +### Root filesystem
>>> +
>>> +A root filesystem depends on:
>>> +
>>> +1. a minimal set of files (the result of `debootstrap`) and
>>> +1. additional files and configurations obtained from *Debian packages*.
>>> +
>>> +The ISAR way to **specify** them is quite complex and involves *ISAR variables* that can be found in configuration files or recipes and artifacts and metadata being provided by *Debian packages*.
>>> +
>>> +See following sections for more information about how to specify the content of the root filesystem:
>>> +
>>> +* [ISAR `IMAGE_INSTALL` variable](#isar-image_install-variable)
>>> +* [ISAR `IMAGE_PREINSTALL` variable](#isar-image_preinstall-variable)
>>> +* [ISAR `DEBIAN_DEPENDS` variable](#isar-debian_depends-variable)
>>> +* [Debian `Depends` control-file value](#debian-depends-control-file-value)
>>> +
>>> +## Buildtime environment specification
>>> +
>>> +The creation of the [above mentioned runtime units](#runtime-environment-specification) (disk images, bootloaders, root filesystems, packages, ...) takes place in different environments.
>>> +
>>> +These environments have to be specified.
>>> +
>>> +### Buildchroot
>>> +
>>> +The *Debian packages* are built in the so-called *buildchroot* filesystem.
>>> +
>>> +It's a Debian chroot that is used to build custom Debian packages required by the **[root filesystem](#root-filesystem)**.
>>> +This build environment depends on the presence of certain artifacts (compilers, configurations, linkers, ...).
>>> +
>>> +The ISAR way to **specify** them is quite complex and involves *ISAR variables* that can be found in configuration files or recipes and artifacts and metadata being provided by *Debian packages*.
>>> +
>>> +See following sections for more information about how to specify the content of the chroot:
>>> +
>>> +* [ISAR `IMAGER_INSTALL` variable](#isar-image_install-variable)
>>> +* [ISAR `BUILDCHROOT_PREINSTALL` variable](#isar-buildchroot_preinstall-variable)
>>> +* [Debian `Build-Depends` control-file value](#debian-build-depends-control-file-value)
>>> +
>>> +If a disk image is being created, the same environment is being used for building the image.
>>> +For this usage additional dependencies might appear. See the [imager](#imager) section for more information.
>>> +
>>> +### Imager
>>> +
>>> +The *disk images* are built in the so-called *imager*.
>>> +
>>> +It's a Debian chroot that is used to build **[disk images](#disk-image)**.
>>> +This build environment depends on the presence of certain artifacts (compilers, configurations, linkers, ...).
>>> +Although conceptually the imager doesn't have anything to do with the buildchroot, in the implementation the same chroot is being used for both tasks.
>>> +
>>> +The ISAR way to **specify** them involves *ISAR variables* that can be found in configuration files or recipes.
>>> +
>>> +See following sections for more information about how to specify the content of the chroot:
>>> +
>>> +* [ISAR `WIC_IMAGER_INSTALL` variable](#isar-wic_imager_install-variable)
>>> +* [ISAR `IMAGER_BUILD_DEPS` variable](#isar-imager_build_deps-variable)
>>> +
>>> +### Target filesystem
>>> +
>>> +It's a Debian root filesystem that results from creating a minimalistic Debian skeleton via `debootstrap` and installing Debian packages on it, resulting in the [root filesystem](#root-
>>> filesystem).
>>> +
>>> +There's no need to **specify** this build environment, since it's the [root filesystem of the runtime environment](#root-filesystem).
>>> +
>>> +## Build process specification
>>> +
>>> +ISAR relies on a subset of Bitbake to take care of the build process.
>>> +Therefore the ISAR way to **specify** the different tasks involved in the process is Bitbake recipes with some *Bitbake variables* (well documented ) and some *ISAR variables* (some of them are
>>> only
>>> syntactic sugar).
>>> +
>>> +See following sections for more information about how to specify inter-tasks dependencies:
>>> +
>>> +* [Bitbake `DEPENDS` variable](#bitbake-depends-variable)
>>> +* [ISAR `IMAGE_INSTALL` variable](#isar-image_install-variable)
>>> +* [ISAR `IMAGER_BUILD_DEPS` variable](#isar-imager_build_deps-variable)
>>> +
>>> +## Mapping: Variables <=> Build environment and Stages
>>> +
>>> +| Variable | Environment | Stage |
>>> +|:-------- |:-----------------:|:-----:|
>>> +| [`DEPENDS`](#bitbake-depends-variable) | | All |
>>> +| [`IMAGE_INSTALL`](#isar-image_install-variable) | [Root filesystem](#root-filesystem) | [Root filesystem](#root-filesystem) preparation and build |
>>> +| [`IMAGE_PREINSTALL`](#isar-image_preinstall-variable) | [Root filesystem](#root-filesystem) | |
>>> +| [`DEBIAN_DEPENDS`](#isar-debian_depends-variable) | [Root filesystem](#root-filesystem) | |
>>> +| [`Depends` control-file](#debian-depends-control-file-value) | [Root filesystem](#root-filesystem) | |
>>> +| [`IMAGE_TRANSIENT_PACKAGES`](#isar-image_transient_packages-variable) | [Root filesystem](#root-filesystem) | [Root filesystem](#root-filesystem) preparation and build |
>>> +| [`BUILDCHROOT_PREINSTALL`](#isar-buildchroot_preinstall-variable) | [Buildchroot](#buildroot) | |
>>> +| [`Build-Depends` control-file](#debian-build-depends-control-file-value) | [Buildchroot](#buildroot) | |
>>> +| [`IMAGER_BUILD_DEPS`](#isar-imager_build_deps-variable) | | [Imager](#imager) preparation |
>>> +| [`IMAGER_INSTALL`](#isar-imager_install-variable) | [Imager](#imager) | |
>>> +| [`WIC_IMAGER_INSTALL`](#isar-wic_imager_install-variable) | [Imager](#imager) | |
>>> +
>>> +## Dependencies specification variables
>>> +
>>> +### Bitbake `DEPENDS` variable
>>> +
>>> +This is a **Bitbake** variable that specifies **Bitbake inter-task dependencies**.
>>> +
>>> +Since it specifies the recipes that another recipe depend upon, this variable can be specified in **package recipes** and **image recipes**.
>>> +
>>> +### ISAR `IMAGE_INSTALL` variable
>>> +
>>> +This is originally a **Bitbake** variable that has conceptually the same goal in ISAR as in Yocto.
>>> +In the case of ISAR it specifies at the same time **Bitbake inter-task dependencies** and **image custom Debian package dependencies**.
>>> +
>>> +It basically specifies that a **[root filesystem](#root-filesystem)** (the prefix 'IMAGE' can be misleading here) requires a custom Debian package.
>>> +But since custom Debian packages are being built from package recipes, this variable also specifies that an image recipe depends on package recipes.
>>> +
>>> +For this "magic" to work both the Debian package and the package recipe have to have the same name.
>>> +Something else than this 1:1 behaviour (e.g. a recipe creating multiple Debian binary packages out of a single Debian source package) requires some hacking out of the scope of this document.
>>> +
>>> +This variable can be found in **configuration files** and **image recipes**.
>>> +
>>> +### ISAR `IMAGE_PREINSTALL` variable
>>> +
>>> +This is an **ISAR-only** variable that specifies **image upstream Debian package dependencies**.
>>> +
>>> +It specifies that a **[root filesystem](#root-filesystem)** requires certain *upstream Debian packages*, being "upstream Debian packages" ready built packages that can be downloaded from a Debian
>>> package repository.
>>> +
>>> +This variable can be found in **configuration files** and **image recipes**.
>>> +
>>> +### ISAR `DEBIAN_DEPENDS` variable
>>> +
>>> +This is an **ISAR-only** variable that specifies **Debian inter-package dependencies** and can be used only on package recipes inheriting from the **`dpkg-raw` class**.
>>> +
>>> +It lists the Debian binary packages that a custom Debian package depends upon.
>>> +It can be used for both upstream and custom packages.
>>> +It results in an entry in the corresponding **[Debian `Depends` control-file value](#debian-depends-control-file-value)**.
>>> +
>>> +A common usage is for creating meta-packages that only keep dependencies on further packages synchronized.
>>> +
>>> +This variable can be found in **package recipes**.
>>> +
>>> +### ISAR `IMAGE_TRANSIENT_PACKAGES` variable
>>> +
>>> +This is an **ISAR-only** variable that specifies **image custom Debian package dependencies** and **Bitbake inter-task dependencies**.
>>> +
>>> +This is a variable that is rarely needed and only for a very specific use case!
>>> +That is when a package is needed in the **[root filesystem](#root-filesystem)** during the **root filesystem build stage**, but not in the final **[root filesystem](#root-filesystem)**.
>>> +Meaning that the installation is only transient, therefore its name.
>>> +
>>> +This variable can be found in **configuration files** and **image recipes**.
>>> +
>>> +### Debian `Depends` control-file value
>>> +
>>> +This is an **Debian** variable that specifies **Debian inter-package dependencies**.
>>> +
>>> +`Depends` specifies the Debian packages that are required in the [runtime environment](#runtime-environment-specification) by the package declaring the dependencies.
>>> +
>>> +It relies on standard Debian mechanisms, therefore the corresponding [official Debian documentation][pkg-rels] provides the best reference.
>>> +
>>> +### ISAR `BUILDCHROOT_PREINSTALL` variable
>>> +
>>> +This is an **ISAR-only** variable that specifies **buildchroot Debian package dependencies**.
>>> +
>>> +It specifies that **[buildchroot](#buildroot)** requires certain *Debian packages*.
>>> +
>>> +This variable can be found in **buildchroot recipes**.
>>> +
>>> +### Debian `Build-Depends` control-file value
>>> +
>>> +This is an **Debian** variable that specifies **Debian inter-package dependencies**.
>>> +
>>> +`Build-Depends` specifies the Debian packages that are required in the [buildtime environment](#buildtime-environment-specification) to build the package declaring the dependencies.
>>> +
>>> +It relies on standard Debian mechanisms, therefore the corresponding [official Debian documentation][pkg-rels] provides the best reference.
>>> +
>>> +### ISAR `IMAGER_BUILD_DEPS` variable
>>> +
>>> +This is an **ISAR-only** variable that specifies **Bitbake inter-task dependencies**.
>>> +
>>> +It has exactly the same effect as `DEPENDS`.
>>> +It simply provides syntactic sugar to separate dependencies on recipes for building the runtime environment from dependencies on recipes for building the **[imager](#imager)**.
>>> +
>>> +This variable can be found in **configuration files**.
>>> +
>>> +### ISAR `IMAGER_INSTALL` variable
>>> +
>>> +This is an **ISAR-only** variable that specifies **imager Debian package dependencies**.
>>> +
>>> +It specifies that certain *Debian packages* have to be installed on the **[imager](#imager)**.
>>> +
>>> +This variable can be found typically in **configuration files**, although it could also be part of **image recipes**.
>>> +
>>> +### ISAR `WIC_IMAGER_INSTALL` variable
>>> +
>>> +This is an **ISAR-only** variable that specifies **imager Debian package dependencies**.
>>> +
>>> +It specifies that an **[imager](#imager)** requires certain *Debian packages* so that WIC can build the disk images (e.g. `parted`).
>>> +
>>> +This variable can be found typically in **configuration files**, although it could also be part of **image recipes**.
>>> +
>>> +[wks]: https://www.yoctoproject.org/docs/current/ref-manual/ref-manual.html#ref-kickstart "OpenEmbedded Kickstart (.wks) Reference"
>>> +[pkg-rels]: https://www.debian.org/doc/debian-policy/ch-relationships.html "Debian policy: Declaring relationships between packages"
>>> --
>>> 2.11.0
>>>
>>
>>
>> --
>> Maxim Osipov
>> ilbers GmbH
>> Maria-Merian-Str. 8
>> 85521 Ottobrunn
>> Germany
>> +49 (151) 6517 6917
>> mosipov@ilbers.de
>> http://ilbers.de/
>> Commercial register Munich, HRB 214197
>> General Manager: Baurzhan Ismagulov
>>
--
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [RFC] v2 Documentation: dependencies specification in Bitbake and Debian
2019-04-03 8:32 ` Maxim Yu. Osipov
@ 2019-04-03 9:06 ` Cirujano Cuesta, Silvano
0 siblings, 0 replies; 8+ messages in thread
From: Cirujano Cuesta, Silvano @ 2019-04-03 9:06 UTC (permalink / raw)
To: mosipov, isar-users
On Wed, 2019-04-03 at 10:32 +0200, Maxim Yu. Osipov wrote:
> Hi Silvano,
>
> On 4/3/19 10:24 AM, Cirujano Cuesta, Silvano wrote:
> > Hi Maxim,
> >
> > Thank you for the information!
> > I'm sorry, in the hurry I just concentrated in the content and not that much in the format of the e-mail.
> > I've just used wrongly my corporate e-mail client (Evolution) and
> > copy&paste.
> > But I know in general how to do it properly both with Evolution [1] and send-email and pay more attention when contributing code intended to be merged.
> >
> > I found a cover letter overkilling and therefore I intended to put my message in the part of the diff being ignored (after '---' and before the files list).
> > The contribution guide mentions a cover letter associated to patch series, but I still find it a bit overkilling for such a RFC.
> >
> > BTW I didn't find any information in the contribution guide about the 'scissors format' for providing some text to be ignored.
> > Is is accepted for single patch contributions?
>
> Sure - send it in 'scissors format' if you find it more appropriate for
> your patch. We are not "strict cover letter" purists :)
Ups, cutting above the 'scissors' is not the default for 'git am' :-/
I'll therefore stick for this RFC and future RFCs and Patches to the established workflow with cover letter.
Regards,
Silvano
>
> Thanks,
> Maxim.
>
> > I still see the advantage of a cover letter for patch series.
> >
> > Cheers,
> > Silvano
> >
> > [1] https://www.kernel.org/doc/html/v5.0/process/email-clients.html?highlight=evolution#evolution-gui
> >
> > On Wed, 2019-04-03 at 09:19 +0200, Maxim Yu. Osipov wrote:
> > > Hi Silvano,
> > >
> > > I've tried to apply your patch but 'git am' complains about malformed line.
> > >
> > > Please have a look in the section Formatting Patches in
> > > https://github.com/ilbers/isar/blob/next/CONTRIBUTING.md
> > >
> > > Personally I use the following sequence of commands when submitting the
> > > patch (f.e. v2) to the mailing list:
> > >
> > > git format-patch --cover-letter --subject-prefix="PATCH v2" -o out -1
> > > vim out/0000-cover-letter.patch
> > > git send-email --suppress-cc=all --to isar-users@googlegroups.com out/*
> > >
> > > In cover letter people usually mention differences versus previous
> > > patch series version and other stuff which shouldn't go into the commit
> > > message.
> > >
> > >
> > > Thanks,
> > > Maxim.
> > >
> > >
> > > On 3/29/19 2:17 PM, Cirujano Cuesta, Silvano wrote:
> > > > Reposting my first message on this thread adding the patch for mailing list review.
> > > >
> > > >
> > > > After my first try a couple of weeks ago, I'm coming back with a new try to improve the documentation.
> > > >
> > > > I've taken Claudius feedback into account (thank you again for your very valuable input!).
> > > > And I let two colleagues of mine with a similar superficial/deep knowledge of ISAR review it.
> > > >
> > > > Right now it's everything in a single MarkDown document, but I'd rather put the section "Dependency specification variables" into a separate document.
> > > > I keep it together right now to simplify the reviews.
> > > >
> > > > For those of you who like to have some comfort in life, I've pushed the document to a fork of ISAR in my personal area.
> > > > That way you can take advantage of MarkDown conversion to HTML and rendering going to [1] ;-)
> > > >
> > > > [1] https://github.com/Silvanoc/isar/blob/silvano/dependencies-documentation/doc/dependencies.md
> > > >
> > > > Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>
> > > > ---
> > > > doc/dependencies.md | 255 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > 1 file changed, 255 insertions(+)
> > > > create mode 100644 doc/dependencies.md
> > > >
> > > > diff --git a/doc/dependencies.md b/doc/dependencies.md
> > > > new file mode 100644
> > > > index 0000000..390e677
> > > > --- /dev/null
> > > > +++ b/doc/dependencies.md
> > > > @@ -0,0 +1,255 @@
> > > > +# Dependency management in ISAR
> > > > +
> > > > +## Introduction
> > > > +
> > > > +The typical goal of an ISAR build is to obtain:
> > > > +
> > > > +1. a disk image containing one or more partition images (all optional),
> > > > +1. containing one or more Debian root filesystems with
> > > > +1. certain Debian packages installed.
> > > > +
> > > > +Reaching this goal relies on the availability of the above listed elements on *runtime*, the specification of the **[runtime environments](#runtime-environment-specification)** helps the
> > > > different
> > > > build tools ensuring the availability of the mentioned elements on runtime.
> > > > +
> > > > +Usually some of the runtime dependencies have to be built, and it happens in special environments. These **[buildtime environments](#buildtime-environment-specification)** help the different
> > > > build
> > > > tools preparing the environment for building the runtime dependencies.
> > > > +
> > > > +The differentiation between buildtime and runtime environments is very important.
> > > > +The fact that ISAR uses Debian both for buildtime and runtime environments might mislead some people.
> > > > +In general the names of the variables being used to specify those dependencies aren't supporting this differentiation, but difficulting it.
> > > > +
> > > > +Not only the build environments have to be specified, but also the **[build process](#build-process-specification)** that takes care of building and integrating both the runtime dependencies
> > > > and
> > > > the
> > > > buildtime dependencies.
> > > > +
> > > > +## Runtime environment specification
> > > > +
> > > > +As mentioned above, ISAR can build [partitioned disk images](#disk-image) or only [root filesystems](#root-filesystem).
> > > > +
> > > > +Let's first analyze how the content of the different runtime containers can be specified from bigger to smaller container.
> > > > +
> > > > +```
> > > > ++---------------------------------------------------------------------------------------------------+
> > > > +| |
> > > > +| DISK IMAGE |
> > > > +| |
> > > > +| +------------+ +-----------------------------------+ +------------------------------+ |
> > > > +| | | | | | || |
> > > > +| | Bootloader | | Bootable partition | | Other partitions ||| |
> > > > +| | | | | | |||| |
> > > > +| | | | +------------------------------+ | | |||| |
> > > > +| | | | | | | | |||| |
> > > > +| | | | | ROOT FILESYSTEM | | | |||| |
> > > > +| +------------+ | | | | | |||| |
> > > > +| | | +-------------------+ | | | |||| |
> > > > +| | | | | | | | | |||| |
> > > > +| | | | DEBIAN PACKAGES | | | | | |||| |
> > > > +| | | | | | | | | |||| |
> > > > +| | | | | | | | | |||| |
> > > > +| | | | | | | | | |||| |
> > > > +| | | +-------------------+ | | | | |||| |
> > > > +| | | ---------------------+ | | | |||| |
> > > > +| | | | | +------------------------------+||| |
> > > > +| | +------------------------------+ | -------------------------------+|| |
> > > > +| | | --------------------------------+| |
> > > > +| +-----------------------------------+ --------------------------------+ |
> > > > +| |
> > > > ++---------------------------------------------------------------------------------------------------+
> > > > +```
> > > > +
> > > > +### Disk image
> > > > +
> > > > +A disk image depends on:
> > > > +
> > > > +1. a bootloader (optional) and
> > > > +1. one or more partitions, containing one of them a [root filesystem](#root-filesystem).
> > > > +
> > > > +The ISAR way to **specify** them is through *WKS files* (OpenEmbedded KickStart files).
> > > > +ISAR doesn't change it in any significant way, therefore the corresponding [Yocto documentation][wks] provides the best reference.
> > > > +
> > > > +### Root filesystem
> > > > +
> > > > +A root filesystem depends on:
> > > > +
> > > > +1. a minimal set of files (the result of `debootstrap`) and
> > > > +1. additional files and configurations obtained from *Debian packages*.
> > > > +
> > > > +The ISAR way to **specify** them is quite complex and involves *ISAR variables* that can be found in configuration files or recipes and artifacts and metadata being provided by *Debian
> > > > packages*.
> > > > +
> > > > +See following sections for more information about how to specify the content of the root filesystem:
> > > > +
> > > > +* [ISAR `IMAGE_INSTALL` variable](#isar-image_install-variable)
> > > > +* [ISAR `IMAGE_PREINSTALL` variable](#isar-image_preinstall-variable)
> > > > +* [ISAR `DEBIAN_DEPENDS` variable](#isar-debian_depends-variable)
> > > > +* [Debian `Depends` control-file value](#debian-depends-control-file-value)
> > > > +
> > > > +## Buildtime environment specification
> > > > +
> > > > +The creation of the [above mentioned runtime units](#runtime-environment-specification) (disk images, bootloaders, root filesystems, packages, ...) takes place in different environments.
> > > > +
> > > > +These environments have to be specified.
> > > > +
> > > > +### Buildchroot
> > > > +
> > > > +The *Debian packages* are built in the so-called *buildchroot* filesystem.
> > > > +
> > > > +It's a Debian chroot that is used to build custom Debian packages required by the **[root filesystem](#root-filesystem)**.
> > > > +This build environment depends on the presence of certain artifacts (compilers, configurations, linkers, ...).
> > > > +
> > > > +The ISAR way to **specify** them is quite complex and involves *ISAR variables* that can be found in configuration files or recipes and artifacts and metadata being provided by *Debian
> > > > packages*.
> > > > +
> > > > +See following sections for more information about how to specify the content of the chroot:
> > > > +
> > > > +* [ISAR `IMAGER_INSTALL` variable](#isar-image_install-variable)
> > > > +* [ISAR `BUILDCHROOT_PREINSTALL` variable](#isar-buildchroot_preinstall-variable)
> > > > +* [Debian `Build-Depends` control-file value](#debian-build-depends-control-file-value)
> > > > +
> > > > +If a disk image is being created, the same environment is being used for building the image.
> > > > +For this usage additional dependencies might appear. See the [imager](#imager) section for more information.
> > > > +
> > > > +### Imager
> > > > +
> > > > +The *disk images* are built in the so-called *imager*.
> > > > +
> > > > +It's a Debian chroot that is used to build **[disk images](#disk-image)**.
> > > > +This build environment depends on the presence of certain artifacts (compilers, configurations, linkers, ...).
> > > > +Although conceptually the imager doesn't have anything to do with the buildchroot, in the implementation the same chroot is being used for both tasks.
> > > > +
> > > > +The ISAR way to **specify** them involves *ISAR variables* that can be found in configuration files or recipes.
> > > > +
> > > > +See following sections for more information about how to specify the content of the chroot:
> > > > +
> > > > +* [ISAR `WIC_IMAGER_INSTALL` variable](#isar-wic_imager_install-variable)
> > > > +* [ISAR `IMAGER_BUILD_DEPS` variable](#isar-imager_build_deps-variable)
> > > > +
> > > > +### Target filesystem
> > > > +
> > > > +It's a Debian root filesystem that results from creating a minimalistic Debian skeleton via `debootstrap` and installing Debian packages on it, resulting in the [root filesystem](#root-
> > > > filesystem).
> > > > +
> > > > +There's no need to **specify** this build environment, since it's the [root filesystem of the runtime environment](#root-filesystem).
> > > > +
> > > > +## Build process specification
> > > > +
> > > > +ISAR relies on a subset of Bitbake to take care of the build process.
> > > > +Therefore the ISAR way to **specify** the different tasks involved in the process is Bitbake recipes with some *Bitbake variables* (well documented ) and some *ISAR variables* (some of them
> > > > are
> > > > only
> > > > syntactic sugar).
> > > > +
> > > > +See following sections for more information about how to specify inter-tasks dependencies:
> > > > +
> > > > +* [Bitbake `DEPENDS` variable](#bitbake-depends-variable)
> > > > +* [ISAR `IMAGE_INSTALL` variable](#isar-image_install-variable)
> > > > +* [ISAR `IMAGER_BUILD_DEPS` variable](#isar-imager_build_deps-variable)
> > > > +
> > > > +## Mapping: Variables <=> Build environment and Stages
> > > > +
> > > > +| Variable | Environment | Stage |
> > > > +|:-------- |:-----------------:|:-----:|
> > > > +| [`DEPENDS`](#bitbake-depends-variable) | | All |
> > > > +| [`IMAGE_INSTALL`](#isar-image_install-variable) | [Root filesystem](#root-filesystem) | [Root filesystem](#root-filesystem) preparation and build |
> > > > +| [`IMAGE_PREINSTALL`](#isar-image_preinstall-variable) | [Root filesystem](#root-filesystem) | |
> > > > +| [`DEBIAN_DEPENDS`](#isar-debian_depends-variable) | [Root filesystem](#root-filesystem) | |
> > > > +| [`Depends` control-file](#debian-depends-control-file-value) | [Root filesystem](#root-filesystem) | |
> > > > +| [`IMAGE_TRANSIENT_PACKAGES`](#isar-image_transient_packages-variable) | [Root filesystem](#root-filesystem) | [Root filesystem](#root-filesystem) preparation and build |
> > > > +| [`BUILDCHROOT_PREINSTALL`](#isar-buildchroot_preinstall-variable) | [Buildchroot](#buildroot) | |
> > > > +| [`Build-Depends` control-file](#debian-build-depends-control-file-value) | [Buildchroot](#buildroot) | |
> > > > +| [`IMAGER_BUILD_DEPS`](#isar-imager_build_deps-variable) | | [Imager](#imager) preparation |
> > > > +| [`IMAGER_INSTALL`](#isar-imager_install-variable) | [Imager](#imager) | |
> > > > +| [`WIC_IMAGER_INSTALL`](#isar-wic_imager_install-variable) | [Imager](#imager) | |
> > > > +
> > > > +## Dependencies specification variables
> > > > +
> > > > +### Bitbake `DEPENDS` variable
> > > > +
> > > > +This is a **Bitbake** variable that specifies **Bitbake inter-task dependencies**.
> > > > +
> > > > +Since it specifies the recipes that another recipe depend upon, this variable can be specified in **package recipes** and **image recipes**.
> > > > +
> > > > +### ISAR `IMAGE_INSTALL` variable
> > > > +
> > > > +This is originally a **Bitbake** variable that has conceptually the same goal in ISAR as in Yocto.
> > > > +In the case of ISAR it specifies at the same time **Bitbake inter-task dependencies** and **image custom Debian package dependencies**.
> > > > +
> > > > +It basically specifies that a **[root filesystem](#root-filesystem)** (the prefix 'IMAGE' can be misleading here) requires a custom Debian package.
> > > > +But since custom Debian packages are being built from package recipes, this variable also specifies that an image recipe depends on package recipes.
> > > > +
> > > > +For this "magic" to work both the Debian package and the package recipe have to have the same name.
> > > > +Something else than this 1:1 behaviour (e.g. a recipe creating multiple Debian binary packages out of a single Debian source package) requires some hacking out of the scope of this document.
> > > > +
> > > > +This variable can be found in **configuration files** and **image recipes**.
> > > > +
> > > > +### ISAR `IMAGE_PREINSTALL` variable
> > > > +
> > > > +This is an **ISAR-only** variable that specifies **image upstream Debian package dependencies**.
> > > > +
> > > > +It specifies that a **[root filesystem](#root-filesystem)** requires certain *upstream Debian packages*, being "upstream Debian packages" ready built packages that can be downloaded from a
> > > > Debian
> > > > package repository.
> > > > +
> > > > +This variable can be found in **configuration files** and **image recipes**.
> > > > +
> > > > +### ISAR `DEBIAN_DEPENDS` variable
> > > > +
> > > > +This is an **ISAR-only** variable that specifies **Debian inter-package dependencies** and can be used only on package recipes inheriting from the **`dpkg-raw` class**.
> > > > +
> > > > +It lists the Debian binary packages that a custom Debian package depends upon.
> > > > +It can be used for both upstream and custom packages.
> > > > +It results in an entry in the corresponding **[Debian `Depends` control-file value](#debian-depends-control-file-value)**.
> > > > +
> > > > +A common usage is for creating meta-packages that only keep dependencies on further packages synchronized.
> > > > +
> > > > +This variable can be found in **package recipes**.
> > > > +
> > > > +### ISAR `IMAGE_TRANSIENT_PACKAGES` variable
> > > > +
> > > > +This is an **ISAR-only** variable that specifies **image custom Debian package dependencies** and **Bitbake inter-task dependencies**.
> > > > +
> > > > +This is a variable that is rarely needed and only for a very specific use case!
> > > > +That is when a package is needed in the **[root filesystem](#root-filesystem)** during the **root filesystem build stage**, but not in the final **[root filesystem](#root-filesystem)**.
> > > > +Meaning that the installation is only transient, therefore its name.
> > > > +
> > > > +This variable can be found in **configuration files** and **image recipes**.
> > > > +
> > > > +### Debian `Depends` control-file value
> > > > +
> > > > +This is an **Debian** variable that specifies **Debian inter-package dependencies**.
> > > > +
> > > > +`Depends` specifies the Debian packages that are required in the [runtime environment](#runtime-environment-specification) by the package declaring the dependencies.
> > > > +
> > > > +It relies on standard Debian mechanisms, therefore the corresponding [official Debian documentation][pkg-rels] provides the best reference.
> > > > +
> > > > +### ISAR `BUILDCHROOT_PREINSTALL` variable
> > > > +
> > > > +This is an **ISAR-only** variable that specifies **buildchroot Debian package dependencies**.
> > > > +
> > > > +It specifies that **[buildchroot](#buildroot)** requires certain *Debian packages*.
> > > > +
> > > > +This variable can be found in **buildchroot recipes**.
> > > > +
> > > > +### Debian `Build-Depends` control-file value
> > > > +
> > > > +This is an **Debian** variable that specifies **Debian inter-package dependencies**.
> > > > +
> > > > +`Build-Depends` specifies the Debian packages that are required in the [buildtime environment](#buildtime-environment-specification) to build the package declaring the dependencies.
> > > > +
> > > > +It relies on standard Debian mechanisms, therefore the corresponding [official Debian documentation][pkg-rels] provides the best reference.
> > > > +
> > > > +### ISAR `IMAGER_BUILD_DEPS` variable
> > > > +
> > > > +This is an **ISAR-only** variable that specifies **Bitbake inter-task dependencies**.
> > > > +
> > > > +It has exactly the same effect as `DEPENDS`.
> > > > +It simply provides syntactic sugar to separate dependencies on recipes for building the runtime environment from dependencies on recipes for building the **[imager](#imager)**.
> > > > +
> > > > +This variable can be found in **configuration files**.
> > > > +
> > > > +### ISAR `IMAGER_INSTALL` variable
> > > > +
> > > > +This is an **ISAR-only** variable that specifies **imager Debian package dependencies**.
> > > > +
> > > > +It specifies that certain *Debian packages* have to be installed on the **[imager](#imager)**.
> > > > +
> > > > +This variable can be found typically in **configuration files**, although it could also be part of **image recipes**.
> > > > +
> > > > +### ISAR `WIC_IMAGER_INSTALL` variable
> > > > +
> > > > +This is an **ISAR-only** variable that specifies **imager Debian package dependencies**.
> > > > +
> > > > +It specifies that an **[imager](#imager)** requires certain *Debian packages* so that WIC can build the disk images (e.g. `parted`).
> > > > +
> > > > +This variable can be found typically in **configuration files**, although it could also be part of **image recipes**.
> > > > +
> > > > +[wks]: https://www.yoctoproject.org/docs/current/ref-manual/ref-manual.html#ref-kickstart "OpenEmbedded Kickstart (.wks) Reference"
> > > > +[pkg-rels]: https://www.debian.org/doc/debian-policy/ch-relationships.html "Debian policy: Declaring relationships between packages"
> > > > --
> > > > 2.11.0
> > > >
> > >
> > >
> > > --
> > > Maxim Osipov
> > > ilbers GmbH
> > > Maria-Merian-Str. 8
> > > 85521 Ottobrunn
> > > Germany
> > > +49 (151) 6517 6917
> > > mosipov@ilbers.de
> > > http://ilbers.de/
> > > Commercial register Munich, HRB 214197
> > > General Manager: Baurzhan Ismagulov
> > >
>
>
--
With best regards,
Silvano Cirujano Cuesta
Siemens AG
Smart Embedded Systems
Mobile: +49 173 3181462
mailto:silvano.cirujano-cuesta@siemens.com
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2019-04-03 9:06 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-02-21 15:33 [RFC] v2 Documentation: dependencies specification in Bitbake and Debian Cirujano Cuesta, Silvano
2019-03-28 18:30 ` Henning Schild
2019-03-29 13:02 ` Cirujano Cuesta, Silvano
2019-03-29 13:17 ` Cirujano Cuesta, Silvano
2019-04-03 7:19 ` Maxim Yu. Osipov
2019-04-03 8:24 ` Cirujano Cuesta, Silvano
2019-04-03 8:32 ` Maxim Yu. Osipov
2019-04-03 9:06 ` Cirujano Cuesta, Silvano
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox