* [RFC] v2 Documentation: dependencies specification in Bitbake and Debian @ 2019-02-21 15:33 Cirujano Cuesta, Silvano 2019-03-28 18:30 ` Henning Schild 2019-03-29 13:17 ` Cirujano Cuesta, Silvano 0 siblings, 2 replies; 8+ messages in thread From: Cirujano Cuesta, Silvano @ 2019-02-21 15:33 UTC (permalink / raw) To: isar-users Hi, After my first try a couple of weeks ago, I'm coming back with a new try to improve the documentation. I've taken Claudius feedback into account (thank you again for your very valuable input!). And I let two colleagues of mine with a similar superficial/deep knowledge of ISAR review it. Right now it's everything in a single MarkDown document, but I'd rather put the section "Dependency specification variables" into a separate document. I keep it together right now to simplify the reviews. For those of you who like to have some comfort in life, I've pushed the document to a fork of ISAR in my personal area. That way you can take advantage of MarkDown conversion to HTML and rendering going to [1] ;-) [1] https://github.com/Silvanoc/isar/blob/silvano/dependencies-documentation/doc/dependencies.md -- With best regards, Silvano Cirujano Cuesta ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [RFC] v2 Documentation: dependencies specification in Bitbake and Debian 2019-02-21 15:33 [RFC] v2 Documentation: dependencies specification in Bitbake and Debian Cirujano Cuesta, Silvano @ 2019-03-28 18:30 ` Henning Schild 2019-03-29 13:02 ` Cirujano Cuesta, Silvano 2019-03-29 13:17 ` Cirujano Cuesta, Silvano 1 sibling, 1 reply; 8+ messages in thread From: Henning Schild @ 2019-03-28 18:30 UTC (permalink / raw) To: [ext] Cirujano Cuesta, Silvano; +Cc: isar-users Am Thu, 21 Feb 2019 15:33:02 +0000 schrieb "[ext] Cirujano Cuesta, Silvano" <silvano.cirujano-cuesta@siemens.com>: > Hi, > > After my first try a couple of weeks ago, I'm coming back with a new > try to improve the documentation. > > I've taken Claudius feedback into account (thank you again for your > very valuable input!). And I let two colleagues of mine with a > similar superficial/deep knowledge of ISAR review it. > > Right now it's everything in a single MarkDown document, but I'd > rather put the section "Dependency specification variables" into a > separate document. I keep it together right now to simplify the > reviews. > > For those of you who like to have some comfort in life, I've pushed > the document to a fork of ISAR in my personal area. That way you can > take advantage of MarkDown conversion to HTML and rendering going to > [1] ;-) Some is ok and since the patch is not on the list, the review is not either. https://github.com/Silvanoc/isar/commit/b3dddefac3004c392edac14b5dd7974f411b0583 Henning > [1] > https://github.com/Silvanoc/isar/blob/silvano/dependencies-documentation/doc/dependencies.md > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [RFC] v2 Documentation: dependencies specification in Bitbake and Debian 2019-03-28 18:30 ` Henning Schild @ 2019-03-29 13:02 ` Cirujano Cuesta, Silvano 0 siblings, 0 replies; 8+ messages in thread From: Cirujano Cuesta, Silvano @ 2019-03-29 13:02 UTC (permalink / raw) To: Schild, Henning; +Cc: isar-users On Thu, 2019-03-28 at 19:30 +0100, Henning Schild wrote: > Am Thu, 21 Feb 2019 15:33:02 +0000 > schrieb "[ext] Cirujano Cuesta, Silvano" > <silvano.cirujano-cuesta@siemens.com>: > > > Hi, > > > > After my first try a couple of weeks ago, I'm coming back with a new > > try to improve the documentation. > > > > I've taken Claudius feedback into account (thank you again for your > > very valuable input!). And I let two colleagues of mine with a > > similar superficial/deep knowledge of ISAR review it. > > > > Right now it's everything in a single MarkDown document, but I'd > > rather put the section "Dependency specification variables" into a > > separate document. I keep it together right now to simplify the > > reviews. > > > > For those of you who like to have some comfort in life, I've pushed > > the document to a fork of ISAR in my personal area. That way you can > > take advantage of MarkDown conversion to HTML and rendering going to > > [1] ;-) > > Some is ok and since the patch is not on the list, the review is not > either. > > https://github.com/Silvanoc/isar/commit/b3dddefac3004c392edac14b5dd7974f411b0583 > > Henning Ups, sorry. Claudius already pointed it out offline, but I forgot to send the patch :-S I don't know how to understand 'some is ok'... I've already considered Hennings comment in Github, but I'll nevertheless send the patch in a separate message for review in the mailing list. BR, Silvano > > > [1] > > https://github.com/Silvanoc/isar/blob/silvano/dependencies-documentation/doc/dependencies.md > > > > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [RFC] v2 Documentation: dependencies specification in Bitbake and Debian 2019-02-21 15:33 [RFC] v2 Documentation: dependencies specification in Bitbake and Debian Cirujano Cuesta, Silvano 2019-03-28 18:30 ` Henning Schild @ 2019-03-29 13:17 ` Cirujano Cuesta, Silvano 2019-04-03 7:19 ` Maxim Yu. Osipov 1 sibling, 1 reply; 8+ messages in thread From: Cirujano Cuesta, Silvano @ 2019-03-29 13:17 UTC (permalink / raw) To: isar-users Reposting my first message on this thread adding the patch for mailing list review. After my first try a couple of weeks ago, I'm coming back with a new try to improve the documentation. I've taken Claudius feedback into account (thank you again for your very valuable input!). And I let two colleagues of mine with a similar superficial/deep knowledge of ISAR review it. Right now it's everything in a single MarkDown document, but I'd rather put the section "Dependency specification variables" into a separate document. I keep it together right now to simplify the reviews. For those of you who like to have some comfort in life, I've pushed the document to a fork of ISAR in my personal area. That way you can take advantage of MarkDown conversion to HTML and rendering going to [1] ;-) [1] https://github.com/Silvanoc/isar/blob/silvano/dependencies-documentation/doc/dependencies.md Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com> --- doc/dependencies.md | 255 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 255 insertions(+) create mode 100644 doc/dependencies.md diff --git a/doc/dependencies.md b/doc/dependencies.md new file mode 100644 index 0000000..390e677 --- /dev/null +++ b/doc/dependencies.md @@ -0,0 +1,255 @@ +# Dependency management in ISAR + +## Introduction + +The typical goal of an ISAR build is to obtain: + +1. a disk image containing one or more partition images (all optional), +1. containing one or more Debian root filesystems with +1. certain Debian packages installed. + +Reaching this goal relies on the availability of the above listed elements on *runtime*, the specification of the **[runtime environments](#runtime-environment-specification)** helps the different build tools ensuring the availability of the mentioned elements on runtime. + +Usually some of the runtime dependencies have to be built, and it happens in special environments. These **[buildtime environments](#buildtime-environment-specification)** help the different build tools preparing the environment for building the runtime dependencies. + +The differentiation between buildtime and runtime environments is very important. +The fact that ISAR uses Debian both for buildtime and runtime environments might mislead some people. +In general the names of the variables being used to specify those dependencies aren't supporting this differentiation, but difficulting it. + +Not only the build environments have to be specified, but also the **[build process](#build-process-specification)** that takes care of building and integrating both the runtime dependencies and the buildtime dependencies. + +## Runtime environment specification + +As mentioned above, ISAR can build [partitioned disk images](#disk-image) or only [root filesystems](#root-filesystem). + +Let's first analyze how the content of the different runtime containers can be specified from bigger to smaller container. + +``` ++---------------------------------------------------------------------------------------------------+ +| | +| DISK IMAGE | +| | +| +------------+ +-----------------------------------+ +------------------------------+ | +| | | | | | || | +| | Bootloader | | Bootable partition | | Other partitions ||| | +| | | | | | |||| | +| | | | +------------------------------+ | | |||| | +| | | | | | | | |||| | +| | | | | ROOT FILESYSTEM | | | |||| | +| +------------+ | | | | | |||| | +| | | +-------------------+ | | | |||| | +| | | | | | | | | |||| | +| | | | DEBIAN PACKAGES | | | | | |||| | +| | | | | | | | | |||| | +| | | | | | | | | |||| | +| | | | | | | | | |||| | +| | | +-------------------+ | | | | |||| | +| | | ---------------------+ | | | |||| | +| | | | | +------------------------------+||| | +| | +------------------------------+ | -------------------------------+|| | +| | | --------------------------------+| | +| +-----------------------------------+ --------------------------------+ | +| | ++---------------------------------------------------------------------------------------------------+ +``` + +### Disk image + +A disk image depends on: + +1. a bootloader (optional) and +1. one or more partitions, containing one of them a [root filesystem](#root-filesystem). + +The ISAR way to **specify** them is through *WKS files* (OpenEmbedded KickStart files). +ISAR doesn't change it in any significant way, therefore the corresponding [Yocto documentation][wks] provides the best reference. + +### Root filesystem + +A root filesystem depends on: + +1. a minimal set of files (the result of `debootstrap`) and +1. additional files and configurations obtained from *Debian packages*. + +The ISAR way to **specify** them is quite complex and involves *ISAR variables* that can be found in configuration files or recipes and artifacts and metadata being provided by *Debian packages*. + +See following sections for more information about how to specify the content of the root filesystem: + +* [ISAR `IMAGE_INSTALL` variable](#isar-image_install-variable) +* [ISAR `IMAGE_PREINSTALL` variable](#isar-image_preinstall-variable) +* [ISAR `DEBIAN_DEPENDS` variable](#isar-debian_depends-variable) +* [Debian `Depends` control-file value](#debian-depends-control-file-value) + +## Buildtime environment specification + +The creation of the [above mentioned runtime units](#runtime-environment-specification) (disk images, bootloaders, root filesystems, packages, ...) takes place in different environments. + +These environments have to be specified. + +### Buildchroot + +The *Debian packages* are built in the so-called *buildchroot* filesystem. + +It's a Debian chroot that is used to build custom Debian packages required by the **[root filesystem](#root-filesystem)**. +This build environment depends on the presence of certain artifacts (compilers, configurations, linkers, ...). + +The ISAR way to **specify** them is quite complex and involves *ISAR variables* that can be found in configuration files or recipes and artifacts and metadata being provided by *Debian packages*. + +See following sections for more information about how to specify the content of the chroot: + +* [ISAR `IMAGER_INSTALL` variable](#isar-image_install-variable) +* [ISAR `BUILDCHROOT_PREINSTALL` variable](#isar-buildchroot_preinstall-variable) +* [Debian `Build-Depends` control-file value](#debian-build-depends-control-file-value) + +If a disk image is being created, the same environment is being used for building the image. +For this usage additional dependencies might appear. See the [imager](#imager) section for more information. + +### Imager + +The *disk images* are built in the so-called *imager*. + +It's a Debian chroot that is used to build **[disk images](#disk-image)**. +This build environment depends on the presence of certain artifacts (compilers, configurations, linkers, ...). +Although conceptually the imager doesn't have anything to do with the buildchroot, in the implementation the same chroot is being used for both tasks. + +The ISAR way to **specify** them involves *ISAR variables* that can be found in configuration files or recipes. + +See following sections for more information about how to specify the content of the chroot: + +* [ISAR `WIC_IMAGER_INSTALL` variable](#isar-wic_imager_install-variable) +* [ISAR `IMAGER_BUILD_DEPS` variable](#isar-imager_build_deps-variable) + +### Target filesystem + +It's a Debian root filesystem that results from creating a minimalistic Debian skeleton via `debootstrap` and installing Debian packages on it, resulting in the [root filesystem](#root-filesystem). + +There's no need to **specify** this build environment, since it's the [root filesystem of the runtime environment](#root-filesystem). + +## Build process specification + +ISAR relies on a subset of Bitbake to take care of the build process. +Therefore the ISAR way to **specify** the different tasks involved in the process is Bitbake recipes with some *Bitbake variables* (well documented ) and some *ISAR variables* (some of them are only syntactic sugar). + +See following sections for more information about how to specify inter-tasks dependencies: + +* [Bitbake `DEPENDS` variable](#bitbake-depends-variable) +* [ISAR `IMAGE_INSTALL` variable](#isar-image_install-variable) +* [ISAR `IMAGER_BUILD_DEPS` variable](#isar-imager_build_deps-variable) + +## Mapping: Variables <=> Build environment and Stages + +| Variable | Environment | Stage | +|:-------- |:-----------------:|:-----:| +| [`DEPENDS`](#bitbake-depends-variable) | | All | +| [`IMAGE_INSTALL`](#isar-image_install-variable) | [Root filesystem](#root-filesystem) | [Root filesystem](#root-filesystem) preparation and build | +| [`IMAGE_PREINSTALL`](#isar-image_preinstall-variable) | [Root filesystem](#root-filesystem) | | +| [`DEBIAN_DEPENDS`](#isar-debian_depends-variable) | [Root filesystem](#root-filesystem) | | +| [`Depends` control-file](#debian-depends-control-file-value) | [Root filesystem](#root-filesystem) | | +| [`IMAGE_TRANSIENT_PACKAGES`](#isar-image_transient_packages-variable) | [Root filesystem](#root-filesystem) | [Root filesystem](#root-filesystem) preparation and build | +| [`BUILDCHROOT_PREINSTALL`](#isar-buildchroot_preinstall-variable) | [Buildchroot](#buildroot) | | +| [`Build-Depends` control-file](#debian-build-depends-control-file-value) | [Buildchroot](#buildroot) | | +| [`IMAGER_BUILD_DEPS`](#isar-imager_build_deps-variable) | | [Imager](#imager) preparation | +| [`IMAGER_INSTALL`](#isar-imager_install-variable) | [Imager](#imager) | | +| [`WIC_IMAGER_INSTALL`](#isar-wic_imager_install-variable) | [Imager](#imager) | | + +## Dependencies specification variables + +### Bitbake `DEPENDS` variable + +This is a **Bitbake** variable that specifies **Bitbake inter-task dependencies**. + +Since it specifies the recipes that another recipe depend upon, this variable can be specified in **package recipes** and **image recipes**. + +### ISAR `IMAGE_INSTALL` variable + +This is originally a **Bitbake** variable that has conceptually the same goal in ISAR as in Yocto. +In the case of ISAR it specifies at the same time **Bitbake inter-task dependencies** and **image custom Debian package dependencies**. + +It basically specifies that a **[root filesystem](#root-filesystem)** (the prefix 'IMAGE' can be misleading here) requires a custom Debian package. +But since custom Debian packages are being built from package recipes, this variable also specifies that an image recipe depends on package recipes. + +For this "magic" to work both the Debian package and the package recipe have to have the same name. +Something else than this 1:1 behaviour (e.g. a recipe creating multiple Debian binary packages out of a single Debian source package) requires some hacking out of the scope of this document. + +This variable can be found in **configuration files** and **image recipes**. + +### ISAR `IMAGE_PREINSTALL` variable + +This is an **ISAR-only** variable that specifies **image upstream Debian package dependencies**. + +It specifies that a **[root filesystem](#root-filesystem)** requires certain *upstream Debian packages*, being "upstream Debian packages" ready built packages that can be downloaded from a Debian package repository. + +This variable can be found in **configuration files** and **image recipes**. + +### ISAR `DEBIAN_DEPENDS` variable + +This is an **ISAR-only** variable that specifies **Debian inter-package dependencies** and can be used only on package recipes inheriting from the **`dpkg-raw` class**. + +It lists the Debian binary packages that a custom Debian package depends upon. +It can be used for both upstream and custom packages. +It results in an entry in the corresponding **[Debian `Depends` control-file value](#debian-depends-control-file-value)**. + +A common usage is for creating meta-packages that only keep dependencies on further packages synchronized. + +This variable can be found in **package recipes**. + +### ISAR `IMAGE_TRANSIENT_PACKAGES` variable + +This is an **ISAR-only** variable that specifies **image custom Debian package dependencies** and **Bitbake inter-task dependencies**. + +This is a variable that is rarely needed and only for a very specific use case! +That is when a package is needed in the **[root filesystem](#root-filesystem)** during the **root filesystem build stage**, but not in the final **[root filesystem](#root-filesystem)**. +Meaning that the installation is only transient, therefore its name. + +This variable can be found in **configuration files** and **image recipes**. + +### Debian `Depends` control-file value + +This is an **Debian** variable that specifies **Debian inter-package dependencies**. + +`Depends` specifies the Debian packages that are required in the [runtime environment](#runtime-environment-specification) by the package declaring the dependencies. + +It relies on standard Debian mechanisms, therefore the corresponding [official Debian documentation][pkg-rels] provides the best reference. + +### ISAR `BUILDCHROOT_PREINSTALL` variable + +This is an **ISAR-only** variable that specifies **buildchroot Debian package dependencies**. + +It specifies that **[buildchroot](#buildroot)** requires certain *Debian packages*. + +This variable can be found in **buildchroot recipes**. + +### Debian `Build-Depends` control-file value + +This is an **Debian** variable that specifies **Debian inter-package dependencies**. + +`Build-Depends` specifies the Debian packages that are required in the [buildtime environment](#buildtime-environment-specification) to build the package declaring the dependencies. + +It relies on standard Debian mechanisms, therefore the corresponding [official Debian documentation][pkg-rels] provides the best reference. + +### ISAR `IMAGER_BUILD_DEPS` variable + +This is an **ISAR-only** variable that specifies **Bitbake inter-task dependencies**. + +It has exactly the same effect as `DEPENDS`. +It simply provides syntactic sugar to separate dependencies on recipes for building the runtime environment from dependencies on recipes for building the **[imager](#imager)**. + +This variable can be found in **configuration files**. + +### ISAR `IMAGER_INSTALL` variable + +This is an **ISAR-only** variable that specifies **imager Debian package dependencies**. + +It specifies that certain *Debian packages* have to be installed on the **[imager](#imager)**. + +This variable can be found typically in **configuration files**, although it could also be part of **image recipes**. + +### ISAR `WIC_IMAGER_INSTALL` variable + +This is an **ISAR-only** variable that specifies **imager Debian package dependencies**. + +It specifies that an **[imager](#imager)** requires certain *Debian packages* so that WIC can build the disk images (e.g. `parted`). + +This variable can be found typically in **configuration files**, although it could also be part of **image recipes**. + +[wks]: https://www.yoctoproject.org/docs/current/ref-manual/ref-manual.html#ref-kickstart "OpenEmbedded Kickstart (.wks) Reference" +[pkg-rels]: https://www.debian.org/doc/debian-policy/ch-relationships.html "Debian policy: Declaring relationships between packages" -- 2.11.0 ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [RFC] v2 Documentation: dependencies specification in Bitbake and Debian 2019-03-29 13:17 ` Cirujano Cuesta, Silvano @ 2019-04-03 7:19 ` Maxim Yu. Osipov 2019-04-03 8:24 ` Cirujano Cuesta, Silvano 0 siblings, 1 reply; 8+ messages in thread From: Maxim Yu. Osipov @ 2019-04-03 7:19 UTC (permalink / raw) To: Cirujano Cuesta, Silvano, isar-users Hi Silvano, I've tried to apply your patch but 'git am' complains about malformed line. Please have a look in the section Formatting Patches in https://github.com/ilbers/isar/blob/next/CONTRIBUTING.md Personally I use the following sequence of commands when submitting the patch (f.e. v2) to the mailing list: git format-patch --cover-letter --subject-prefix="PATCH v2" -o out -1 vim out/0000-cover-letter.patch git send-email --suppress-cc=all --to isar-users@googlegroups.com out/* In cover letter people usually mention differences versus previous patch series version and other stuff which shouldn't go into the commit message. Thanks, Maxim. On 3/29/19 2:17 PM, Cirujano Cuesta, Silvano wrote: > Reposting my first message on this thread adding the patch for mailing list review. > > > After my first try a couple of weeks ago, I'm coming back with a new try to improve the documentation. > > I've taken Claudius feedback into account (thank you again for your very valuable input!). > And I let two colleagues of mine with a similar superficial/deep knowledge of ISAR review it. > > Right now it's everything in a single MarkDown document, but I'd rather put the section "Dependency specification variables" into a separate document. > I keep it together right now to simplify the reviews. > > For those of you who like to have some comfort in life, I've pushed the document to a fork of ISAR in my personal area. > That way you can take advantage of MarkDown conversion to HTML and rendering going to [1] ;-) > > [1] https://github.com/Silvanoc/isar/blob/silvano/dependencies-documentation/doc/dependencies.md > > Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com> > --- > doc/dependencies.md | 255 ++++++++++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 255 insertions(+) > create mode 100644 doc/dependencies.md > > diff --git a/doc/dependencies.md b/doc/dependencies.md > new file mode 100644 > index 0000000..390e677 > --- /dev/null > +++ b/doc/dependencies.md > @@ -0,0 +1,255 @@ > +# Dependency management in ISAR > + > +## Introduction > + > +The typical goal of an ISAR build is to obtain: > + > +1. a disk image containing one or more partition images (all optional), > +1. containing one or more Debian root filesystems with > +1. certain Debian packages installed. > + > +Reaching this goal relies on the availability of the above listed elements on *runtime*, the specification of the **[runtime environments](#runtime-environment-specification)** helps the different > build tools ensuring the availability of the mentioned elements on runtime. > + > +Usually some of the runtime dependencies have to be built, and it happens in special environments. These **[buildtime environments](#buildtime-environment-specification)** help the different build > tools preparing the environment for building the runtime dependencies. > + > +The differentiation between buildtime and runtime environments is very important. > +The fact that ISAR uses Debian both for buildtime and runtime environments might mislead some people. > +In general the names of the variables being used to specify those dependencies aren't supporting this differentiation, but difficulting it. > + > +Not only the build environments have to be specified, but also the **[build process](#build-process-specification)** that takes care of building and integrating both the runtime dependencies and the > buildtime dependencies. > + > +## Runtime environment specification > + > +As mentioned above, ISAR can build [partitioned disk images](#disk-image) or only [root filesystems](#root-filesystem). > + > +Let's first analyze how the content of the different runtime containers can be specified from bigger to smaller container. > + > +``` > ++---------------------------------------------------------------------------------------------------+ > +| | > +| DISK IMAGE | > +| | > +| +------------+ +-----------------------------------+ +------------------------------+ | > +| | | | | | || | > +| | Bootloader | | Bootable partition | | Other partitions ||| | > +| | | | | | |||| | > +| | | | +------------------------------+ | | |||| | > +| | | | | | | | |||| | > +| | | | | ROOT FILESYSTEM | | | |||| | > +| +------------+ | | | | | |||| | > +| | | +-------------------+ | | | |||| | > +| | | | | | | | | |||| | > +| | | | DEBIAN PACKAGES | | | | | |||| | > +| | | | | | | | | |||| | > +| | | | | | | | | |||| | > +| | | | | | | | | |||| | > +| | | +-------------------+ | | | | |||| | > +| | | ---------------------+ | | | |||| | > +| | | | | +------------------------------+||| | > +| | +------------------------------+ | -------------------------------+|| | > +| | | --------------------------------+| | > +| +-----------------------------------+ --------------------------------+ | > +| | > ++---------------------------------------------------------------------------------------------------+ > +``` > + > +### Disk image > + > +A disk image depends on: > + > +1. a bootloader (optional) and > +1. one or more partitions, containing one of them a [root filesystem](#root-filesystem). > + > +The ISAR way to **specify** them is through *WKS files* (OpenEmbedded KickStart files). > +ISAR doesn't change it in any significant way, therefore the corresponding [Yocto documentation][wks] provides the best reference. > + > +### Root filesystem > + > +A root filesystem depends on: > + > +1. a minimal set of files (the result of `debootstrap`) and > +1. additional files and configurations obtained from *Debian packages*. > + > +The ISAR way to **specify** them is quite complex and involves *ISAR variables* that can be found in configuration files or recipes and artifacts and metadata being provided by *Debian packages*. > + > +See following sections for more information about how to specify the content of the root filesystem: > + > +* [ISAR `IMAGE_INSTALL` variable](#isar-image_install-variable) > +* [ISAR `IMAGE_PREINSTALL` variable](#isar-image_preinstall-variable) > +* [ISAR `DEBIAN_DEPENDS` variable](#isar-debian_depends-variable) > +* [Debian `Depends` control-file value](#debian-depends-control-file-value) > + > +## Buildtime environment specification > + > +The creation of the [above mentioned runtime units](#runtime-environment-specification) (disk images, bootloaders, root filesystems, packages, ...) takes place in different environments. > + > +These environments have to be specified. > + > +### Buildchroot > + > +The *Debian packages* are built in the so-called *buildchroot* filesystem. > + > +It's a Debian chroot that is used to build custom Debian packages required by the **[root filesystem](#root-filesystem)**. > +This build environment depends on the presence of certain artifacts (compilers, configurations, linkers, ...). > + > +The ISAR way to **specify** them is quite complex and involves *ISAR variables* that can be found in configuration files or recipes and artifacts and metadata being provided by *Debian packages*. > + > +See following sections for more information about how to specify the content of the chroot: > + > +* [ISAR `IMAGER_INSTALL` variable](#isar-image_install-variable) > +* [ISAR `BUILDCHROOT_PREINSTALL` variable](#isar-buildchroot_preinstall-variable) > +* [Debian `Build-Depends` control-file value](#debian-build-depends-control-file-value) > + > +If a disk image is being created, the same environment is being used for building the image. > +For this usage additional dependencies might appear. See the [imager](#imager) section for more information. > + > +### Imager > + > +The *disk images* are built in the so-called *imager*. > + > +It's a Debian chroot that is used to build **[disk images](#disk-image)**. > +This build environment depends on the presence of certain artifacts (compilers, configurations, linkers, ...). > +Although conceptually the imager doesn't have anything to do with the buildchroot, in the implementation the same chroot is being used for both tasks. > + > +The ISAR way to **specify** them involves *ISAR variables* that can be found in configuration files or recipes. > + > +See following sections for more information about how to specify the content of the chroot: > + > +* [ISAR `WIC_IMAGER_INSTALL` variable](#isar-wic_imager_install-variable) > +* [ISAR `IMAGER_BUILD_DEPS` variable](#isar-imager_build_deps-variable) > + > +### Target filesystem > + > +It's a Debian root filesystem that results from creating a minimalistic Debian skeleton via `debootstrap` and installing Debian packages on it, resulting in the [root filesystem](#root-filesystem). > + > +There's no need to **specify** this build environment, since it's the [root filesystem of the runtime environment](#root-filesystem). > + > +## Build process specification > + > +ISAR relies on a subset of Bitbake to take care of the build process. > +Therefore the ISAR way to **specify** the different tasks involved in the process is Bitbake recipes with some *Bitbake variables* (well documented ) and some *ISAR variables* (some of them are only > syntactic sugar). > + > +See following sections for more information about how to specify inter-tasks dependencies: > + > +* [Bitbake `DEPENDS` variable](#bitbake-depends-variable) > +* [ISAR `IMAGE_INSTALL` variable](#isar-image_install-variable) > +* [ISAR `IMAGER_BUILD_DEPS` variable](#isar-imager_build_deps-variable) > + > +## Mapping: Variables <=> Build environment and Stages > + > +| Variable | Environment | Stage | > +|:-------- |:-----------------:|:-----:| > +| [`DEPENDS`](#bitbake-depends-variable) | | All | > +| [`IMAGE_INSTALL`](#isar-image_install-variable) | [Root filesystem](#root-filesystem) | [Root filesystem](#root-filesystem) preparation and build | > +| [`IMAGE_PREINSTALL`](#isar-image_preinstall-variable) | [Root filesystem](#root-filesystem) | | > +| [`DEBIAN_DEPENDS`](#isar-debian_depends-variable) | [Root filesystem](#root-filesystem) | | > +| [`Depends` control-file](#debian-depends-control-file-value) | [Root filesystem](#root-filesystem) | | > +| [`IMAGE_TRANSIENT_PACKAGES`](#isar-image_transient_packages-variable) | [Root filesystem](#root-filesystem) | [Root filesystem](#root-filesystem) preparation and build | > +| [`BUILDCHROOT_PREINSTALL`](#isar-buildchroot_preinstall-variable) | [Buildchroot](#buildroot) | | > +| [`Build-Depends` control-file](#debian-build-depends-control-file-value) | [Buildchroot](#buildroot) | | > +| [`IMAGER_BUILD_DEPS`](#isar-imager_build_deps-variable) | | [Imager](#imager) preparation | > +| [`IMAGER_INSTALL`](#isar-imager_install-variable) | [Imager](#imager) | | > +| [`WIC_IMAGER_INSTALL`](#isar-wic_imager_install-variable) | [Imager](#imager) | | > + > +## Dependencies specification variables > + > +### Bitbake `DEPENDS` variable > + > +This is a **Bitbake** variable that specifies **Bitbake inter-task dependencies**. > + > +Since it specifies the recipes that another recipe depend upon, this variable can be specified in **package recipes** and **image recipes**. > + > +### ISAR `IMAGE_INSTALL` variable > + > +This is originally a **Bitbake** variable that has conceptually the same goal in ISAR as in Yocto. > +In the case of ISAR it specifies at the same time **Bitbake inter-task dependencies** and **image custom Debian package dependencies**. > + > +It basically specifies that a **[root filesystem](#root-filesystem)** (the prefix 'IMAGE' can be misleading here) requires a custom Debian package. > +But since custom Debian packages are being built from package recipes, this variable also specifies that an image recipe depends on package recipes. > + > +For this "magic" to work both the Debian package and the package recipe have to have the same name. > +Something else than this 1:1 behaviour (e.g. a recipe creating multiple Debian binary packages out of a single Debian source package) requires some hacking out of the scope of this document. > + > +This variable can be found in **configuration files** and **image recipes**. > + > +### ISAR `IMAGE_PREINSTALL` variable > + > +This is an **ISAR-only** variable that specifies **image upstream Debian package dependencies**. > + > +It specifies that a **[root filesystem](#root-filesystem)** requires certain *upstream Debian packages*, being "upstream Debian packages" ready built packages that can be downloaded from a Debian > package repository. > + > +This variable can be found in **configuration files** and **image recipes**. > + > +### ISAR `DEBIAN_DEPENDS` variable > + > +This is an **ISAR-only** variable that specifies **Debian inter-package dependencies** and can be used only on package recipes inheriting from the **`dpkg-raw` class**. > + > +It lists the Debian binary packages that a custom Debian package depends upon. > +It can be used for both upstream and custom packages. > +It results in an entry in the corresponding **[Debian `Depends` control-file value](#debian-depends-control-file-value)**. > + > +A common usage is for creating meta-packages that only keep dependencies on further packages synchronized. > + > +This variable can be found in **package recipes**. > + > +### ISAR `IMAGE_TRANSIENT_PACKAGES` variable > + > +This is an **ISAR-only** variable that specifies **image custom Debian package dependencies** and **Bitbake inter-task dependencies**. > + > +This is a variable that is rarely needed and only for a very specific use case! > +That is when a package is needed in the **[root filesystem](#root-filesystem)** during the **root filesystem build stage**, but not in the final **[root filesystem](#root-filesystem)**. > +Meaning that the installation is only transient, therefore its name. > + > +This variable can be found in **configuration files** and **image recipes**. > + > +### Debian `Depends` control-file value > + > +This is an **Debian** variable that specifies **Debian inter-package dependencies**. > + > +`Depends` specifies the Debian packages that are required in the [runtime environment](#runtime-environment-specification) by the package declaring the dependencies. > + > +It relies on standard Debian mechanisms, therefore the corresponding [official Debian documentation][pkg-rels] provides the best reference. > + > +### ISAR `BUILDCHROOT_PREINSTALL` variable > + > +This is an **ISAR-only** variable that specifies **buildchroot Debian package dependencies**. > + > +It specifies that **[buildchroot](#buildroot)** requires certain *Debian packages*. > + > +This variable can be found in **buildchroot recipes**. > + > +### Debian `Build-Depends` control-file value > + > +This is an **Debian** variable that specifies **Debian inter-package dependencies**. > + > +`Build-Depends` specifies the Debian packages that are required in the [buildtime environment](#buildtime-environment-specification) to build the package declaring the dependencies. > + > +It relies on standard Debian mechanisms, therefore the corresponding [official Debian documentation][pkg-rels] provides the best reference. > + > +### ISAR `IMAGER_BUILD_DEPS` variable > + > +This is an **ISAR-only** variable that specifies **Bitbake inter-task dependencies**. > + > +It has exactly the same effect as `DEPENDS`. > +It simply provides syntactic sugar to separate dependencies on recipes for building the runtime environment from dependencies on recipes for building the **[imager](#imager)**. > + > +This variable can be found in **configuration files**. > + > +### ISAR `IMAGER_INSTALL` variable > + > +This is an **ISAR-only** variable that specifies **imager Debian package dependencies**. > + > +It specifies that certain *Debian packages* have to be installed on the **[imager](#imager)**. > + > +This variable can be found typically in **configuration files**, although it could also be part of **image recipes**. > + > +### ISAR `WIC_IMAGER_INSTALL` variable > + > +This is an **ISAR-only** variable that specifies **imager Debian package dependencies**. > + > +It specifies that an **[imager](#imager)** requires certain *Debian packages* so that WIC can build the disk images (e.g. `parted`). > + > +This variable can be found typically in **configuration files**, although it could also be part of **image recipes**. > + > +[wks]: https://www.yoctoproject.org/docs/current/ref-manual/ref-manual.html#ref-kickstart "OpenEmbedded Kickstart (.wks) Reference" > +[pkg-rels]: https://www.debian.org/doc/debian-policy/ch-relationships.html "Debian policy: Declaring relationships between packages" > -- > 2.11.0 > -- Maxim Osipov ilbers GmbH Maria-Merian-Str. 8 85521 Ottobrunn Germany +49 (151) 6517 6917 mosipov@ilbers.de http://ilbers.de/ Commercial register Munich, HRB 214197 General Manager: Baurzhan Ismagulov ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [RFC] v2 Documentation: dependencies specification in Bitbake and Debian 2019-04-03 7:19 ` Maxim Yu. Osipov @ 2019-04-03 8:24 ` Cirujano Cuesta, Silvano 2019-04-03 8:32 ` Maxim Yu. Osipov 0 siblings, 1 reply; 8+ messages in thread From: Cirujano Cuesta, Silvano @ 2019-04-03 8:24 UTC (permalink / raw) To: mosipov, isar-users Hi Maxim, Thank you for the information! I'm sorry, in the hurry I just concentrated in the content and not that much in the format of the e-mail. I've just used wrongly my corporate e-mail client (Evolution) and copy&paste. But I know in general how to do it properly both with Evolution [1] and send-email and pay more attention when contributing code intended to be merged. I found a cover letter overkilling and therefore I intended to put my message in the part of the diff being ignored (after '---' and before the files list). The contribution guide mentions a cover letter associated to patch series, but I still find it a bit overkilling for such a RFC. BTW I didn't find any information in the contribution guide about the 'scissors format' for providing some text to be ignored. Is is accepted for single patch contributions? I still see the advantage of a cover letter for patch series. Cheers, Silvano [1] https://www.kernel.org/doc/html/v5.0/process/email-clients.html?highlight=evolution#evolution-gui On Wed, 2019-04-03 at 09:19 +0200, Maxim Yu. Osipov wrote: > Hi Silvano, > > I've tried to apply your patch but 'git am' complains about malformed line. > > Please have a look in the section Formatting Patches in > https://github.com/ilbers/isar/blob/next/CONTRIBUTING.md > > Personally I use the following sequence of commands when submitting the > patch (f.e. v2) to the mailing list: > > git format-patch --cover-letter --subject-prefix="PATCH v2" -o out -1 > vim out/0000-cover-letter.patch > git send-email --suppress-cc=all --to isar-users@googlegroups.com out/* > > In cover letter people usually mention differences versus previous > patch series version and other stuff which shouldn't go into the commit > message. > > > Thanks, > Maxim. > > > On 3/29/19 2:17 PM, Cirujano Cuesta, Silvano wrote: > > Reposting my first message on this thread adding the patch for mailing list review. > > > > > > After my first try a couple of weeks ago, I'm coming back with a new try to improve the documentation. > > > > I've taken Claudius feedback into account (thank you again for your very valuable input!). > > And I let two colleagues of mine with a similar superficial/deep knowledge of ISAR review it. > > > > Right now it's everything in a single MarkDown document, but I'd rather put the section "Dependency specification variables" into a separate document. > > I keep it together right now to simplify the reviews. > > > > For those of you who like to have some comfort in life, I've pushed the document to a fork of ISAR in my personal area. > > That way you can take advantage of MarkDown conversion to HTML and rendering going to [1] ;-) > > > > [1] https://github.com/Silvanoc/isar/blob/silvano/dependencies-documentation/doc/dependencies.md > > > > Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com> > > --- > > doc/dependencies.md | 255 ++++++++++++++++++++++++++++++++++++++++++++++++++++ > > 1 file changed, 255 insertions(+) > > create mode 100644 doc/dependencies.md > > > > diff --git a/doc/dependencies.md b/doc/dependencies.md > > new file mode 100644 > > index 0000000..390e677 > > --- /dev/null > > +++ b/doc/dependencies.md > > @@ -0,0 +1,255 @@ > > +# Dependency management in ISAR > > + > > +## Introduction > > + > > +The typical goal of an ISAR build is to obtain: > > + > > +1. a disk image containing one or more partition images (all optional), > > +1. containing one or more Debian root filesystems with > > +1. certain Debian packages installed. > > + > > +Reaching this goal relies on the availability of the above listed elements on *runtime*, the specification of the **[runtime environments](#runtime-environment-specification)** helps the > > different > > build tools ensuring the availability of the mentioned elements on runtime. > > + > > +Usually some of the runtime dependencies have to be built, and it happens in special environments. These **[buildtime environments](#buildtime-environment-specification)** help the different > > build > > tools preparing the environment for building the runtime dependencies. > > + > > +The differentiation between buildtime and runtime environments is very important. > > +The fact that ISAR uses Debian both for buildtime and runtime environments might mislead some people. > > +In general the names of the variables being used to specify those dependencies aren't supporting this differentiation, but difficulting it. > > + > > +Not only the build environments have to be specified, but also the **[build process](#build-process-specification)** that takes care of building and integrating both the runtime dependencies and > > the > > buildtime dependencies. > > + > > +## Runtime environment specification > > + > > +As mentioned above, ISAR can build [partitioned disk images](#disk-image) or only [root filesystems](#root-filesystem). > > + > > +Let's first analyze how the content of the different runtime containers can be specified from bigger to smaller container. > > + > > +``` > > ++---------------------------------------------------------------------------------------------------+ > > +| | > > +| DISK IMAGE | > > +| | > > +| +------------+ +-----------------------------------+ +------------------------------+ | > > +| | | | | | || | > > +| | Bootloader | | Bootable partition | | Other partitions ||| | > > +| | | | | | |||| | > > +| | | | +------------------------------+ | | |||| | > > +| | | | | | | | |||| | > > +| | | | | ROOT FILESYSTEM | | | |||| | > > +| +------------+ | | | | | |||| | > > +| | | +-------------------+ | | | |||| | > > +| | | | | | | | | |||| | > > +| | | | DEBIAN PACKAGES | | | | | |||| | > > +| | | | | | | | | |||| | > > +| | | | | | | | | |||| | > > +| | | | | | | | | |||| | > > +| | | +-------------------+ | | | | |||| | > > +| | | ---------------------+ | | | |||| | > > +| | | | | +------------------------------+||| | > > +| | +------------------------------+ | -------------------------------+|| | > > +| | | --------------------------------+| | > > +| +-----------------------------------+ --------------------------------+ | > > +| | > > ++---------------------------------------------------------------------------------------------------+ > > +``` > > + > > +### Disk image > > + > > +A disk image depends on: > > + > > +1. a bootloader (optional) and > > +1. one or more partitions, containing one of them a [root filesystem](#root-filesystem). > > + > > +The ISAR way to **specify** them is through *WKS files* (OpenEmbedded KickStart files). > > +ISAR doesn't change it in any significant way, therefore the corresponding [Yocto documentation][wks] provides the best reference. > > + > > +### Root filesystem > > + > > +A root filesystem depends on: > > + > > +1. a minimal set of files (the result of `debootstrap`) and > > +1. additional files and configurations obtained from *Debian packages*. > > + > > +The ISAR way to **specify** them is quite complex and involves *ISAR variables* that can be found in configuration files or recipes and artifacts and metadata being provided by *Debian packages*. > > + > > +See following sections for more information about how to specify the content of the root filesystem: > > + > > +* [ISAR `IMAGE_INSTALL` variable](#isar-image_install-variable) > > +* [ISAR `IMAGE_PREINSTALL` variable](#isar-image_preinstall-variable) > > +* [ISAR `DEBIAN_DEPENDS` variable](#isar-debian_depends-variable) > > +* [Debian `Depends` control-file value](#debian-depends-control-file-value) > > + > > +## Buildtime environment specification > > + > > +The creation of the [above mentioned runtime units](#runtime-environment-specification) (disk images, bootloaders, root filesystems, packages, ...) takes place in different environments. > > + > > +These environments have to be specified. > > + > > +### Buildchroot > > + > > +The *Debian packages* are built in the so-called *buildchroot* filesystem. > > + > > +It's a Debian chroot that is used to build custom Debian packages required by the **[root filesystem](#root-filesystem)**. > > +This build environment depends on the presence of certain artifacts (compilers, configurations, linkers, ...). > > + > > +The ISAR way to **specify** them is quite complex and involves *ISAR variables* that can be found in configuration files or recipes and artifacts and metadata being provided by *Debian packages*. > > + > > +See following sections for more information about how to specify the content of the chroot: > > + > > +* [ISAR `IMAGER_INSTALL` variable](#isar-image_install-variable) > > +* [ISAR `BUILDCHROOT_PREINSTALL` variable](#isar-buildchroot_preinstall-variable) > > +* [Debian `Build-Depends` control-file value](#debian-build-depends-control-file-value) > > + > > +If a disk image is being created, the same environment is being used for building the image. > > +For this usage additional dependencies might appear. See the [imager](#imager) section for more information. > > + > > +### Imager > > + > > +The *disk images* are built in the so-called *imager*. > > + > > +It's a Debian chroot that is used to build **[disk images](#disk-image)**. > > +This build environment depends on the presence of certain artifacts (compilers, configurations, linkers, ...). > > +Although conceptually the imager doesn't have anything to do with the buildchroot, in the implementation the same chroot is being used for both tasks. > > + > > +The ISAR way to **specify** them involves *ISAR variables* that can be found in configuration files or recipes. > > + > > +See following sections for more information about how to specify the content of the chroot: > > + > > +* [ISAR `WIC_IMAGER_INSTALL` variable](#isar-wic_imager_install-variable) > > +* [ISAR `IMAGER_BUILD_DEPS` variable](#isar-imager_build_deps-variable) > > + > > +### Target filesystem > > + > > +It's a Debian root filesystem that results from creating a minimalistic Debian skeleton via `debootstrap` and installing Debian packages on it, resulting in the [root filesystem](#root- > > filesystem). > > + > > +There's no need to **specify** this build environment, since it's the [root filesystem of the runtime environment](#root-filesystem). > > + > > +## Build process specification > > + > > +ISAR relies on a subset of Bitbake to take care of the build process. > > +Therefore the ISAR way to **specify** the different tasks involved in the process is Bitbake recipes with some *Bitbake variables* (well documented ) and some *ISAR variables* (some of them are > > only > > syntactic sugar). > > + > > +See following sections for more information about how to specify inter-tasks dependencies: > > + > > +* [Bitbake `DEPENDS` variable](#bitbake-depends-variable) > > +* [ISAR `IMAGE_INSTALL` variable](#isar-image_install-variable) > > +* [ISAR `IMAGER_BUILD_DEPS` variable](#isar-imager_build_deps-variable) > > + > > +## Mapping: Variables <=> Build environment and Stages > > + > > +| Variable | Environment | Stage | > > +|:-------- |:-----------------:|:-----:| > > +| [`DEPENDS`](#bitbake-depends-variable) | | All | > > +| [`IMAGE_INSTALL`](#isar-image_install-variable) | [Root filesystem](#root-filesystem) | [Root filesystem](#root-filesystem) preparation and build | > > +| [`IMAGE_PREINSTALL`](#isar-image_preinstall-variable) | [Root filesystem](#root-filesystem) | | > > +| [`DEBIAN_DEPENDS`](#isar-debian_depends-variable) | [Root filesystem](#root-filesystem) | | > > +| [`Depends` control-file](#debian-depends-control-file-value) | [Root filesystem](#root-filesystem) | | > > +| [`IMAGE_TRANSIENT_PACKAGES`](#isar-image_transient_packages-variable) | [Root filesystem](#root-filesystem) | [Root filesystem](#root-filesystem) preparation and build | > > +| [`BUILDCHROOT_PREINSTALL`](#isar-buildchroot_preinstall-variable) | [Buildchroot](#buildroot) | | > > +| [`Build-Depends` control-file](#debian-build-depends-control-file-value) | [Buildchroot](#buildroot) | | > > +| [`IMAGER_BUILD_DEPS`](#isar-imager_build_deps-variable) | | [Imager](#imager) preparation | > > +| [`IMAGER_INSTALL`](#isar-imager_install-variable) | [Imager](#imager) | | > > +| [`WIC_IMAGER_INSTALL`](#isar-wic_imager_install-variable) | [Imager](#imager) | | > > + > > +## Dependencies specification variables > > + > > +### Bitbake `DEPENDS` variable > > + > > +This is a **Bitbake** variable that specifies **Bitbake inter-task dependencies**. > > + > > +Since it specifies the recipes that another recipe depend upon, this variable can be specified in **package recipes** and **image recipes**. > > + > > +### ISAR `IMAGE_INSTALL` variable > > + > > +This is originally a **Bitbake** variable that has conceptually the same goal in ISAR as in Yocto. > > +In the case of ISAR it specifies at the same time **Bitbake inter-task dependencies** and **image custom Debian package dependencies**. > > + > > +It basically specifies that a **[root filesystem](#root-filesystem)** (the prefix 'IMAGE' can be misleading here) requires a custom Debian package. > > +But since custom Debian packages are being built from package recipes, this variable also specifies that an image recipe depends on package recipes. > > + > > +For this "magic" to work both the Debian package and the package recipe have to have the same name. > > +Something else than this 1:1 behaviour (e.g. a recipe creating multiple Debian binary packages out of a single Debian source package) requires some hacking out of the scope of this document. > > + > > +This variable can be found in **configuration files** and **image recipes**. > > + > > +### ISAR `IMAGE_PREINSTALL` variable > > + > > +This is an **ISAR-only** variable that specifies **image upstream Debian package dependencies**. > > + > > +It specifies that a **[root filesystem](#root-filesystem)** requires certain *upstream Debian packages*, being "upstream Debian packages" ready built packages that can be downloaded from a Debian > > package repository. > > + > > +This variable can be found in **configuration files** and **image recipes**. > > + > > +### ISAR `DEBIAN_DEPENDS` variable > > + > > +This is an **ISAR-only** variable that specifies **Debian inter-package dependencies** and can be used only on package recipes inheriting from the **`dpkg-raw` class**. > > + > > +It lists the Debian binary packages that a custom Debian package depends upon. > > +It can be used for both upstream and custom packages. > > +It results in an entry in the corresponding **[Debian `Depends` control-file value](#debian-depends-control-file-value)**. > > + > > +A common usage is for creating meta-packages that only keep dependencies on further packages synchronized. > > + > > +This variable can be found in **package recipes**. > > + > > +### ISAR `IMAGE_TRANSIENT_PACKAGES` variable > > + > > +This is an **ISAR-only** variable that specifies **image custom Debian package dependencies** and **Bitbake inter-task dependencies**. > > + > > +This is a variable that is rarely needed and only for a very specific use case! > > +That is when a package is needed in the **[root filesystem](#root-filesystem)** during the **root filesystem build stage**, but not in the final **[root filesystem](#root-filesystem)**. > > +Meaning that the installation is only transient, therefore its name. > > + > > +This variable can be found in **configuration files** and **image recipes**. > > + > > +### Debian `Depends` control-file value > > + > > +This is an **Debian** variable that specifies **Debian inter-package dependencies**. > > + > > +`Depends` specifies the Debian packages that are required in the [runtime environment](#runtime-environment-specification) by the package declaring the dependencies. > > + > > +It relies on standard Debian mechanisms, therefore the corresponding [official Debian documentation][pkg-rels] provides the best reference. > > + > > +### ISAR `BUILDCHROOT_PREINSTALL` variable > > + > > +This is an **ISAR-only** variable that specifies **buildchroot Debian package dependencies**. > > + > > +It specifies that **[buildchroot](#buildroot)** requires certain *Debian packages*. > > + > > +This variable can be found in **buildchroot recipes**. > > + > > +### Debian `Build-Depends` control-file value > > + > > +This is an **Debian** variable that specifies **Debian inter-package dependencies**. > > + > > +`Build-Depends` specifies the Debian packages that are required in the [buildtime environment](#buildtime-environment-specification) to build the package declaring the dependencies. > > + > > +It relies on standard Debian mechanisms, therefore the corresponding [official Debian documentation][pkg-rels] provides the best reference. > > + > > +### ISAR `IMAGER_BUILD_DEPS` variable > > + > > +This is an **ISAR-only** variable that specifies **Bitbake inter-task dependencies**. > > + > > +It has exactly the same effect as `DEPENDS`. > > +It simply provides syntactic sugar to separate dependencies on recipes for building the runtime environment from dependencies on recipes for building the **[imager](#imager)**. > > + > > +This variable can be found in **configuration files**. > > + > > +### ISAR `IMAGER_INSTALL` variable > > + > > +This is an **ISAR-only** variable that specifies **imager Debian package dependencies**. > > + > > +It specifies that certain *Debian packages* have to be installed on the **[imager](#imager)**. > > + > > +This variable can be found typically in **configuration files**, although it could also be part of **image recipes**. > > + > > +### ISAR `WIC_IMAGER_INSTALL` variable > > + > > +This is an **ISAR-only** variable that specifies **imager Debian package dependencies**. > > + > > +It specifies that an **[imager](#imager)** requires certain *Debian packages* so that WIC can build the disk images (e.g. `parted`). > > + > > +This variable can be found typically in **configuration files**, although it could also be part of **image recipes**. > > + > > +[wks]: https://www.yoctoproject.org/docs/current/ref-manual/ref-manual.html#ref-kickstart "OpenEmbedded Kickstart (.wks) Reference" > > +[pkg-rels]: https://www.debian.org/doc/debian-policy/ch-relationships.html "Debian policy: Declaring relationships between packages" > > -- > > 2.11.0 > > > > > -- > Maxim Osipov > ilbers GmbH > Maria-Merian-Str. 8 > 85521 Ottobrunn > Germany > +49 (151) 6517 6917 > mosipov@ilbers.de > http://ilbers.de/ > Commercial register Munich, HRB 214197 > General Manager: Baurzhan Ismagulov > -- With best regards, Silvano Cirujano Cuesta Siemens AG Smart Embedded Systems Mobile: +49 173 3181462 mailto:silvano.cirujano-cuesta@siemens.com ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [RFC] v2 Documentation: dependencies specification in Bitbake and Debian 2019-04-03 8:24 ` Cirujano Cuesta, Silvano @ 2019-04-03 8:32 ` Maxim Yu. Osipov 2019-04-03 9:06 ` Cirujano Cuesta, Silvano 0 siblings, 1 reply; 8+ messages in thread From: Maxim Yu. Osipov @ 2019-04-03 8:32 UTC (permalink / raw) To: Cirujano Cuesta, Silvano, isar-users Hi Silvano, On 4/3/19 10:24 AM, Cirujano Cuesta, Silvano wrote: > Hi Maxim, > > Thank you for the information! > I'm sorry, in the hurry I just concentrated in the content and not that much in the format of the e-mail. > I've just used wrongly my corporate e-mail client (Evolution) and > copy&paste. > But I know in general how to do it properly both with Evolution [1] and send-email and pay more attention when contributing code intended to be merged. > > I found a cover letter overkilling and therefore I intended to put my message in the part of the diff being ignored (after '---' and before the files list). > The contribution guide mentions a cover letter associated to patch series, but I still find it a bit overkilling for such a RFC. > > BTW I didn't find any information in the contribution guide about the 'scissors format' for providing some text to be ignored. > Is is accepted for single patch contributions? Sure - send it in 'scissors format' if you find it more appropriate for your patch. We are not "strict cover letter" purists :) Thanks, Maxim. > I still see the advantage of a cover letter for patch series. > > Cheers, > Silvano > > [1] https://www.kernel.org/doc/html/v5.0/process/email-clients.html?highlight=evolution#evolution-gui > > On Wed, 2019-04-03 at 09:19 +0200, Maxim Yu. Osipov wrote: >> Hi Silvano, >> >> I've tried to apply your patch but 'git am' complains about malformed line. >> >> Please have a look in the section Formatting Patches in >> https://github.com/ilbers/isar/blob/next/CONTRIBUTING.md >> >> Personally I use the following sequence of commands when submitting the >> patch (f.e. v2) to the mailing list: >> >> git format-patch --cover-letter --subject-prefix="PATCH v2" -o out -1 >> vim out/0000-cover-letter.patch >> git send-email --suppress-cc=all --to isar-users@googlegroups.com out/* >> >> In cover letter people usually mention differences versus previous >> patch series version and other stuff which shouldn't go into the commit >> message. >> >> >> Thanks, >> Maxim. >> >> >> On 3/29/19 2:17 PM, Cirujano Cuesta, Silvano wrote: >>> Reposting my first message on this thread adding the patch for mailing list review. >>> >>> >>> After my first try a couple of weeks ago, I'm coming back with a new try to improve the documentation. >>> >>> I've taken Claudius feedback into account (thank you again for your very valuable input!). >>> And I let two colleagues of mine with a similar superficial/deep knowledge of ISAR review it. >>> >>> Right now it's everything in a single MarkDown document, but I'd rather put the section "Dependency specification variables" into a separate document. >>> I keep it together right now to simplify the reviews. >>> >>> For those of you who like to have some comfort in life, I've pushed the document to a fork of ISAR in my personal area. >>> That way you can take advantage of MarkDown conversion to HTML and rendering going to [1] ;-) >>> >>> [1] https://github.com/Silvanoc/isar/blob/silvano/dependencies-documentation/doc/dependencies.md >>> >>> Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com> >>> --- >>> doc/dependencies.md | 255 ++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> 1 file changed, 255 insertions(+) >>> create mode 100644 doc/dependencies.md >>> >>> diff --git a/doc/dependencies.md b/doc/dependencies.md >>> new file mode 100644 >>> index 0000000..390e677 >>> --- /dev/null >>> +++ b/doc/dependencies.md >>> @@ -0,0 +1,255 @@ >>> +# Dependency management in ISAR >>> + >>> +## Introduction >>> + >>> +The typical goal of an ISAR build is to obtain: >>> + >>> +1. a disk image containing one or more partition images (all optional), >>> +1. containing one or more Debian root filesystems with >>> +1. certain Debian packages installed. >>> + >>> +Reaching this goal relies on the availability of the above listed elements on *runtime*, the specification of the **[runtime environments](#runtime-environment-specification)** helps the >>> different >>> build tools ensuring the availability of the mentioned elements on runtime. >>> + >>> +Usually some of the runtime dependencies have to be built, and it happens in special environments. These **[buildtime environments](#buildtime-environment-specification)** help the different >>> build >>> tools preparing the environment for building the runtime dependencies. >>> + >>> +The differentiation between buildtime and runtime environments is very important. >>> +The fact that ISAR uses Debian both for buildtime and runtime environments might mislead some people. >>> +In general the names of the variables being used to specify those dependencies aren't supporting this differentiation, but difficulting it. >>> + >>> +Not only the build environments have to be specified, but also the **[build process](#build-process-specification)** that takes care of building and integrating both the runtime dependencies and >>> the >>> buildtime dependencies. >>> + >>> +## Runtime environment specification >>> + >>> +As mentioned above, ISAR can build [partitioned disk images](#disk-image) or only [root filesystems](#root-filesystem). >>> + >>> +Let's first analyze how the content of the different runtime containers can be specified from bigger to smaller container. >>> + >>> +``` >>> ++---------------------------------------------------------------------------------------------------+ >>> +| | >>> +| DISK IMAGE | >>> +| | >>> +| +------------+ +-----------------------------------+ +------------------------------+ | >>> +| | | | | | || | >>> +| | Bootloader | | Bootable partition | | Other partitions ||| | >>> +| | | | | | |||| | >>> +| | | | +------------------------------+ | | |||| | >>> +| | | | | | | | |||| | >>> +| | | | | ROOT FILESYSTEM | | | |||| | >>> +| +------------+ | | | | | |||| | >>> +| | | +-------------------+ | | | |||| | >>> +| | | | | | | | | |||| | >>> +| | | | DEBIAN PACKAGES | | | | | |||| | >>> +| | | | | | | | | |||| | >>> +| | | | | | | | | |||| | >>> +| | | | | | | | | |||| | >>> +| | | +-------------------+ | | | | |||| | >>> +| | | ---------------------+ | | | |||| | >>> +| | | | | +------------------------------+||| | >>> +| | +------------------------------+ | -------------------------------+|| | >>> +| | | --------------------------------+| | >>> +| +-----------------------------------+ --------------------------------+ | >>> +| | >>> ++---------------------------------------------------------------------------------------------------+ >>> +``` >>> + >>> +### Disk image >>> + >>> +A disk image depends on: >>> + >>> +1. a bootloader (optional) and >>> +1. one or more partitions, containing one of them a [root filesystem](#root-filesystem). >>> + >>> +The ISAR way to **specify** them is through *WKS files* (OpenEmbedded KickStart files). >>> +ISAR doesn't change it in any significant way, therefore the corresponding [Yocto documentation][wks] provides the best reference. >>> + >>> +### Root filesystem >>> + >>> +A root filesystem depends on: >>> + >>> +1. a minimal set of files (the result of `debootstrap`) and >>> +1. additional files and configurations obtained from *Debian packages*. >>> + >>> +The ISAR way to **specify** them is quite complex and involves *ISAR variables* that can be found in configuration files or recipes and artifacts and metadata being provided by *Debian packages*. >>> + >>> +See following sections for more information about how to specify the content of the root filesystem: >>> + >>> +* [ISAR `IMAGE_INSTALL` variable](#isar-image_install-variable) >>> +* [ISAR `IMAGE_PREINSTALL` variable](#isar-image_preinstall-variable) >>> +* [ISAR `DEBIAN_DEPENDS` variable](#isar-debian_depends-variable) >>> +* [Debian `Depends` control-file value](#debian-depends-control-file-value) >>> + >>> +## Buildtime environment specification >>> + >>> +The creation of the [above mentioned runtime units](#runtime-environment-specification) (disk images, bootloaders, root filesystems, packages, ...) takes place in different environments. >>> + >>> +These environments have to be specified. >>> + >>> +### Buildchroot >>> + >>> +The *Debian packages* are built in the so-called *buildchroot* filesystem. >>> + >>> +It's a Debian chroot that is used to build custom Debian packages required by the **[root filesystem](#root-filesystem)**. >>> +This build environment depends on the presence of certain artifacts (compilers, configurations, linkers, ...). >>> + >>> +The ISAR way to **specify** them is quite complex and involves *ISAR variables* that can be found in configuration files or recipes and artifacts and metadata being provided by *Debian packages*. >>> + >>> +See following sections for more information about how to specify the content of the chroot: >>> + >>> +* [ISAR `IMAGER_INSTALL` variable](#isar-image_install-variable) >>> +* [ISAR `BUILDCHROOT_PREINSTALL` variable](#isar-buildchroot_preinstall-variable) >>> +* [Debian `Build-Depends` control-file value](#debian-build-depends-control-file-value) >>> + >>> +If a disk image is being created, the same environment is being used for building the image. >>> +For this usage additional dependencies might appear. See the [imager](#imager) section for more information. >>> + >>> +### Imager >>> + >>> +The *disk images* are built in the so-called *imager*. >>> + >>> +It's a Debian chroot that is used to build **[disk images](#disk-image)**. >>> +This build environment depends on the presence of certain artifacts (compilers, configurations, linkers, ...). >>> +Although conceptually the imager doesn't have anything to do with the buildchroot, in the implementation the same chroot is being used for both tasks. >>> + >>> +The ISAR way to **specify** them involves *ISAR variables* that can be found in configuration files or recipes. >>> + >>> +See following sections for more information about how to specify the content of the chroot: >>> + >>> +* [ISAR `WIC_IMAGER_INSTALL` variable](#isar-wic_imager_install-variable) >>> +* [ISAR `IMAGER_BUILD_DEPS` variable](#isar-imager_build_deps-variable) >>> + >>> +### Target filesystem >>> + >>> +It's a Debian root filesystem that results from creating a minimalistic Debian skeleton via `debootstrap` and installing Debian packages on it, resulting in the [root filesystem](#root- >>> filesystem). >>> + >>> +There's no need to **specify** this build environment, since it's the [root filesystem of the runtime environment](#root-filesystem). >>> + >>> +## Build process specification >>> + >>> +ISAR relies on a subset of Bitbake to take care of the build process. >>> +Therefore the ISAR way to **specify** the different tasks involved in the process is Bitbake recipes with some *Bitbake variables* (well documented ) and some *ISAR variables* (some of them are >>> only >>> syntactic sugar). >>> + >>> +See following sections for more information about how to specify inter-tasks dependencies: >>> + >>> +* [Bitbake `DEPENDS` variable](#bitbake-depends-variable) >>> +* [ISAR `IMAGE_INSTALL` variable](#isar-image_install-variable) >>> +* [ISAR `IMAGER_BUILD_DEPS` variable](#isar-imager_build_deps-variable) >>> + >>> +## Mapping: Variables <=> Build environment and Stages >>> + >>> +| Variable | Environment | Stage | >>> +|:-------- |:-----------------:|:-----:| >>> +| [`DEPENDS`](#bitbake-depends-variable) | | All | >>> +| [`IMAGE_INSTALL`](#isar-image_install-variable) | [Root filesystem](#root-filesystem) | [Root filesystem](#root-filesystem) preparation and build | >>> +| [`IMAGE_PREINSTALL`](#isar-image_preinstall-variable) | [Root filesystem](#root-filesystem) | | >>> +| [`DEBIAN_DEPENDS`](#isar-debian_depends-variable) | [Root filesystem](#root-filesystem) | | >>> +| [`Depends` control-file](#debian-depends-control-file-value) | [Root filesystem](#root-filesystem) | | >>> +| [`IMAGE_TRANSIENT_PACKAGES`](#isar-image_transient_packages-variable) | [Root filesystem](#root-filesystem) | [Root filesystem](#root-filesystem) preparation and build | >>> +| [`BUILDCHROOT_PREINSTALL`](#isar-buildchroot_preinstall-variable) | [Buildchroot](#buildroot) | | >>> +| [`Build-Depends` control-file](#debian-build-depends-control-file-value) | [Buildchroot](#buildroot) | | >>> +| [`IMAGER_BUILD_DEPS`](#isar-imager_build_deps-variable) | | [Imager](#imager) preparation | >>> +| [`IMAGER_INSTALL`](#isar-imager_install-variable) | [Imager](#imager) | | >>> +| [`WIC_IMAGER_INSTALL`](#isar-wic_imager_install-variable) | [Imager](#imager) | | >>> + >>> +## Dependencies specification variables >>> + >>> +### Bitbake `DEPENDS` variable >>> + >>> +This is a **Bitbake** variable that specifies **Bitbake inter-task dependencies**. >>> + >>> +Since it specifies the recipes that another recipe depend upon, this variable can be specified in **package recipes** and **image recipes**. >>> + >>> +### ISAR `IMAGE_INSTALL` variable >>> + >>> +This is originally a **Bitbake** variable that has conceptually the same goal in ISAR as in Yocto. >>> +In the case of ISAR it specifies at the same time **Bitbake inter-task dependencies** and **image custom Debian package dependencies**. >>> + >>> +It basically specifies that a **[root filesystem](#root-filesystem)** (the prefix 'IMAGE' can be misleading here) requires a custom Debian package. >>> +But since custom Debian packages are being built from package recipes, this variable also specifies that an image recipe depends on package recipes. >>> + >>> +For this "magic" to work both the Debian package and the package recipe have to have the same name. >>> +Something else than this 1:1 behaviour (e.g. a recipe creating multiple Debian binary packages out of a single Debian source package) requires some hacking out of the scope of this document. >>> + >>> +This variable can be found in **configuration files** and **image recipes**. >>> + >>> +### ISAR `IMAGE_PREINSTALL` variable >>> + >>> +This is an **ISAR-only** variable that specifies **image upstream Debian package dependencies**. >>> + >>> +It specifies that a **[root filesystem](#root-filesystem)** requires certain *upstream Debian packages*, being "upstream Debian packages" ready built packages that can be downloaded from a Debian >>> package repository. >>> + >>> +This variable can be found in **configuration files** and **image recipes**. >>> + >>> +### ISAR `DEBIAN_DEPENDS` variable >>> + >>> +This is an **ISAR-only** variable that specifies **Debian inter-package dependencies** and can be used only on package recipes inheriting from the **`dpkg-raw` class**. >>> + >>> +It lists the Debian binary packages that a custom Debian package depends upon. >>> +It can be used for both upstream and custom packages. >>> +It results in an entry in the corresponding **[Debian `Depends` control-file value](#debian-depends-control-file-value)**. >>> + >>> +A common usage is for creating meta-packages that only keep dependencies on further packages synchronized. >>> + >>> +This variable can be found in **package recipes**. >>> + >>> +### ISAR `IMAGE_TRANSIENT_PACKAGES` variable >>> + >>> +This is an **ISAR-only** variable that specifies **image custom Debian package dependencies** and **Bitbake inter-task dependencies**. >>> + >>> +This is a variable that is rarely needed and only for a very specific use case! >>> +That is when a package is needed in the **[root filesystem](#root-filesystem)** during the **root filesystem build stage**, but not in the final **[root filesystem](#root-filesystem)**. >>> +Meaning that the installation is only transient, therefore its name. >>> + >>> +This variable can be found in **configuration files** and **image recipes**. >>> + >>> +### Debian `Depends` control-file value >>> + >>> +This is an **Debian** variable that specifies **Debian inter-package dependencies**. >>> + >>> +`Depends` specifies the Debian packages that are required in the [runtime environment](#runtime-environment-specification) by the package declaring the dependencies. >>> + >>> +It relies on standard Debian mechanisms, therefore the corresponding [official Debian documentation][pkg-rels] provides the best reference. >>> + >>> +### ISAR `BUILDCHROOT_PREINSTALL` variable >>> + >>> +This is an **ISAR-only** variable that specifies **buildchroot Debian package dependencies**. >>> + >>> +It specifies that **[buildchroot](#buildroot)** requires certain *Debian packages*. >>> + >>> +This variable can be found in **buildchroot recipes**. >>> + >>> +### Debian `Build-Depends` control-file value >>> + >>> +This is an **Debian** variable that specifies **Debian inter-package dependencies**. >>> + >>> +`Build-Depends` specifies the Debian packages that are required in the [buildtime environment](#buildtime-environment-specification) to build the package declaring the dependencies. >>> + >>> +It relies on standard Debian mechanisms, therefore the corresponding [official Debian documentation][pkg-rels] provides the best reference. >>> + >>> +### ISAR `IMAGER_BUILD_DEPS` variable >>> + >>> +This is an **ISAR-only** variable that specifies **Bitbake inter-task dependencies**. >>> + >>> +It has exactly the same effect as `DEPENDS`. >>> +It simply provides syntactic sugar to separate dependencies on recipes for building the runtime environment from dependencies on recipes for building the **[imager](#imager)**. >>> + >>> +This variable can be found in **configuration files**. >>> + >>> +### ISAR `IMAGER_INSTALL` variable >>> + >>> +This is an **ISAR-only** variable that specifies **imager Debian package dependencies**. >>> + >>> +It specifies that certain *Debian packages* have to be installed on the **[imager](#imager)**. >>> + >>> +This variable can be found typically in **configuration files**, although it could also be part of **image recipes**. >>> + >>> +### ISAR `WIC_IMAGER_INSTALL` variable >>> + >>> +This is an **ISAR-only** variable that specifies **imager Debian package dependencies**. >>> + >>> +It specifies that an **[imager](#imager)** requires certain *Debian packages* so that WIC can build the disk images (e.g. `parted`). >>> + >>> +This variable can be found typically in **configuration files**, although it could also be part of **image recipes**. >>> + >>> +[wks]: https://www.yoctoproject.org/docs/current/ref-manual/ref-manual.html#ref-kickstart "OpenEmbedded Kickstart (.wks) Reference" >>> +[pkg-rels]: https://www.debian.org/doc/debian-policy/ch-relationships.html "Debian policy: Declaring relationships between packages" >>> -- >>> 2.11.0 >>> >> >> >> -- >> Maxim Osipov >> ilbers GmbH >> Maria-Merian-Str. 8 >> 85521 Ottobrunn >> Germany >> +49 (151) 6517 6917 >> mosipov@ilbers.de >> http://ilbers.de/ >> Commercial register Munich, HRB 214197 >> General Manager: Baurzhan Ismagulov >> -- Maxim Osipov ilbers GmbH Maria-Merian-Str. 8 85521 Ottobrunn Germany +49 (151) 6517 6917 mosipov@ilbers.de http://ilbers.de/ Commercial register Munich, HRB 214197 General Manager: Baurzhan Ismagulov ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [RFC] v2 Documentation: dependencies specification in Bitbake and Debian 2019-04-03 8:32 ` Maxim Yu. Osipov @ 2019-04-03 9:06 ` Cirujano Cuesta, Silvano 0 siblings, 0 replies; 8+ messages in thread From: Cirujano Cuesta, Silvano @ 2019-04-03 9:06 UTC (permalink / raw) To: mosipov, isar-users On Wed, 2019-04-03 at 10:32 +0200, Maxim Yu. Osipov wrote: > Hi Silvano, > > On 4/3/19 10:24 AM, Cirujano Cuesta, Silvano wrote: > > Hi Maxim, > > > > Thank you for the information! > > I'm sorry, in the hurry I just concentrated in the content and not that much in the format of the e-mail. > > I've just used wrongly my corporate e-mail client (Evolution) and > > copy&paste. > > But I know in general how to do it properly both with Evolution [1] and send-email and pay more attention when contributing code intended to be merged. > > > > I found a cover letter overkilling and therefore I intended to put my message in the part of the diff being ignored (after '---' and before the files list). > > The contribution guide mentions a cover letter associated to patch series, but I still find it a bit overkilling for such a RFC. > > > > BTW I didn't find any information in the contribution guide about the 'scissors format' for providing some text to be ignored. > > Is is accepted for single patch contributions? > > Sure - send it in 'scissors format' if you find it more appropriate for > your patch. We are not "strict cover letter" purists :) Ups, cutting above the 'scissors' is not the default for 'git am' :-/ I'll therefore stick for this RFC and future RFCs and Patches to the established workflow with cover letter. Regards, Silvano > > Thanks, > Maxim. > > > I still see the advantage of a cover letter for patch series. > > > > Cheers, > > Silvano > > > > [1] https://www.kernel.org/doc/html/v5.0/process/email-clients.html?highlight=evolution#evolution-gui > > > > On Wed, 2019-04-03 at 09:19 +0200, Maxim Yu. Osipov wrote: > > > Hi Silvano, > > > > > > I've tried to apply your patch but 'git am' complains about malformed line. > > > > > > Please have a look in the section Formatting Patches in > > > https://github.com/ilbers/isar/blob/next/CONTRIBUTING.md > > > > > > Personally I use the following sequence of commands when submitting the > > > patch (f.e. v2) to the mailing list: > > > > > > git format-patch --cover-letter --subject-prefix="PATCH v2" -o out -1 > > > vim out/0000-cover-letter.patch > > > git send-email --suppress-cc=all --to isar-users@googlegroups.com out/* > > > > > > In cover letter people usually mention differences versus previous > > > patch series version and other stuff which shouldn't go into the commit > > > message. > > > > > > > > > Thanks, > > > Maxim. > > > > > > > > > On 3/29/19 2:17 PM, Cirujano Cuesta, Silvano wrote: > > > > Reposting my first message on this thread adding the patch for mailing list review. > > > > > > > > > > > > After my first try a couple of weeks ago, I'm coming back with a new try to improve the documentation. > > > > > > > > I've taken Claudius feedback into account (thank you again for your very valuable input!). > > > > And I let two colleagues of mine with a similar superficial/deep knowledge of ISAR review it. > > > > > > > > Right now it's everything in a single MarkDown document, but I'd rather put the section "Dependency specification variables" into a separate document. > > > > I keep it together right now to simplify the reviews. > > > > > > > > For those of you who like to have some comfort in life, I've pushed the document to a fork of ISAR in my personal area. > > > > That way you can take advantage of MarkDown conversion to HTML and rendering going to [1] ;-) > > > > > > > > [1] https://github.com/Silvanoc/isar/blob/silvano/dependencies-documentation/doc/dependencies.md > > > > > > > > Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com> > > > > --- > > > > doc/dependencies.md | 255 ++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > 1 file changed, 255 insertions(+) > > > > create mode 100644 doc/dependencies.md > > > > > > > > diff --git a/doc/dependencies.md b/doc/dependencies.md > > > > new file mode 100644 > > > > index 0000000..390e677 > > > > --- /dev/null > > > > +++ b/doc/dependencies.md > > > > @@ -0,0 +1,255 @@ > > > > +# Dependency management in ISAR > > > > + > > > > +## Introduction > > > > + > > > > +The typical goal of an ISAR build is to obtain: > > > > + > > > > +1. a disk image containing one or more partition images (all optional), > > > > +1. containing one or more Debian root filesystems with > > > > +1. certain Debian packages installed. > > > > + > > > > +Reaching this goal relies on the availability of the above listed elements on *runtime*, the specification of the **[runtime environments](#runtime-environment-specification)** helps the > > > > different > > > > build tools ensuring the availability of the mentioned elements on runtime. > > > > + > > > > +Usually some of the runtime dependencies have to be built, and it happens in special environments. These **[buildtime environments](#buildtime-environment-specification)** help the different > > > > build > > > > tools preparing the environment for building the runtime dependencies. > > > > + > > > > +The differentiation between buildtime and runtime environments is very important. > > > > +The fact that ISAR uses Debian both for buildtime and runtime environments might mislead some people. > > > > +In general the names of the variables being used to specify those dependencies aren't supporting this differentiation, but difficulting it. > > > > + > > > > +Not only the build environments have to be specified, but also the **[build process](#build-process-specification)** that takes care of building and integrating both the runtime dependencies > > > > and > > > > the > > > > buildtime dependencies. > > > > + > > > > +## Runtime environment specification > > > > + > > > > +As mentioned above, ISAR can build [partitioned disk images](#disk-image) or only [root filesystems](#root-filesystem). > > > > + > > > > +Let's first analyze how the content of the different runtime containers can be specified from bigger to smaller container. > > > > + > > > > +``` > > > > ++---------------------------------------------------------------------------------------------------+ > > > > +| | > > > > +| DISK IMAGE | > > > > +| | > > > > +| +------------+ +-----------------------------------+ +------------------------------+ | > > > > +| | | | | | || | > > > > +| | Bootloader | | Bootable partition | | Other partitions ||| | > > > > +| | | | | | |||| | > > > > +| | | | +------------------------------+ | | |||| | > > > > +| | | | | | | | |||| | > > > > +| | | | | ROOT FILESYSTEM | | | |||| | > > > > +| +------------+ | | | | | |||| | > > > > +| | | +-------------------+ | | | |||| | > > > > +| | | | | | | | | |||| | > > > > +| | | | DEBIAN PACKAGES | | | | | |||| | > > > > +| | | | | | | | | |||| | > > > > +| | | | | | | | | |||| | > > > > +| | | | | | | | | |||| | > > > > +| | | +-------------------+ | | | | |||| | > > > > +| | | ---------------------+ | | | |||| | > > > > +| | | | | +------------------------------+||| | > > > > +| | +------------------------------+ | -------------------------------+|| | > > > > +| | | --------------------------------+| | > > > > +| +-----------------------------------+ --------------------------------+ | > > > > +| | > > > > ++---------------------------------------------------------------------------------------------------+ > > > > +``` > > > > + > > > > +### Disk image > > > > + > > > > +A disk image depends on: > > > > + > > > > +1. a bootloader (optional) and > > > > +1. one or more partitions, containing one of them a [root filesystem](#root-filesystem). > > > > + > > > > +The ISAR way to **specify** them is through *WKS files* (OpenEmbedded KickStart files). > > > > +ISAR doesn't change it in any significant way, therefore the corresponding [Yocto documentation][wks] provides the best reference. > > > > + > > > > +### Root filesystem > > > > + > > > > +A root filesystem depends on: > > > > + > > > > +1. a minimal set of files (the result of `debootstrap`) and > > > > +1. additional files and configurations obtained from *Debian packages*. > > > > + > > > > +The ISAR way to **specify** them is quite complex and involves *ISAR variables* that can be found in configuration files or recipes and artifacts and metadata being provided by *Debian > > > > packages*. > > > > + > > > > +See following sections for more information about how to specify the content of the root filesystem: > > > > + > > > > +* [ISAR `IMAGE_INSTALL` variable](#isar-image_install-variable) > > > > +* [ISAR `IMAGE_PREINSTALL` variable](#isar-image_preinstall-variable) > > > > +* [ISAR `DEBIAN_DEPENDS` variable](#isar-debian_depends-variable) > > > > +* [Debian `Depends` control-file value](#debian-depends-control-file-value) > > > > + > > > > +## Buildtime environment specification > > > > + > > > > +The creation of the [above mentioned runtime units](#runtime-environment-specification) (disk images, bootloaders, root filesystems, packages, ...) takes place in different environments. > > > > + > > > > +These environments have to be specified. > > > > + > > > > +### Buildchroot > > > > + > > > > +The *Debian packages* are built in the so-called *buildchroot* filesystem. > > > > + > > > > +It's a Debian chroot that is used to build custom Debian packages required by the **[root filesystem](#root-filesystem)**. > > > > +This build environment depends on the presence of certain artifacts (compilers, configurations, linkers, ...). > > > > + > > > > +The ISAR way to **specify** them is quite complex and involves *ISAR variables* that can be found in configuration files or recipes and artifacts and metadata being provided by *Debian > > > > packages*. > > > > + > > > > +See following sections for more information about how to specify the content of the chroot: > > > > + > > > > +* [ISAR `IMAGER_INSTALL` variable](#isar-image_install-variable) > > > > +* [ISAR `BUILDCHROOT_PREINSTALL` variable](#isar-buildchroot_preinstall-variable) > > > > +* [Debian `Build-Depends` control-file value](#debian-build-depends-control-file-value) > > > > + > > > > +If a disk image is being created, the same environment is being used for building the image. > > > > +For this usage additional dependencies might appear. See the [imager](#imager) section for more information. > > > > + > > > > +### Imager > > > > + > > > > +The *disk images* are built in the so-called *imager*. > > > > + > > > > +It's a Debian chroot that is used to build **[disk images](#disk-image)**. > > > > +This build environment depends on the presence of certain artifacts (compilers, configurations, linkers, ...). > > > > +Although conceptually the imager doesn't have anything to do with the buildchroot, in the implementation the same chroot is being used for both tasks. > > > > + > > > > +The ISAR way to **specify** them involves *ISAR variables* that can be found in configuration files or recipes. > > > > + > > > > +See following sections for more information about how to specify the content of the chroot: > > > > + > > > > +* [ISAR `WIC_IMAGER_INSTALL` variable](#isar-wic_imager_install-variable) > > > > +* [ISAR `IMAGER_BUILD_DEPS` variable](#isar-imager_build_deps-variable) > > > > + > > > > +### Target filesystem > > > > + > > > > +It's a Debian root filesystem that results from creating a minimalistic Debian skeleton via `debootstrap` and installing Debian packages on it, resulting in the [root filesystem](#root- > > > > filesystem). > > > > + > > > > +There's no need to **specify** this build environment, since it's the [root filesystem of the runtime environment](#root-filesystem). > > > > + > > > > +## Build process specification > > > > + > > > > +ISAR relies on a subset of Bitbake to take care of the build process. > > > > +Therefore the ISAR way to **specify** the different tasks involved in the process is Bitbake recipes with some *Bitbake variables* (well documented ) and some *ISAR variables* (some of them > > > > are > > > > only > > > > syntactic sugar). > > > > + > > > > +See following sections for more information about how to specify inter-tasks dependencies: > > > > + > > > > +* [Bitbake `DEPENDS` variable](#bitbake-depends-variable) > > > > +* [ISAR `IMAGE_INSTALL` variable](#isar-image_install-variable) > > > > +* [ISAR `IMAGER_BUILD_DEPS` variable](#isar-imager_build_deps-variable) > > > > + > > > > +## Mapping: Variables <=> Build environment and Stages > > > > + > > > > +| Variable | Environment | Stage | > > > > +|:-------- |:-----------------:|:-----:| > > > > +| [`DEPENDS`](#bitbake-depends-variable) | | All | > > > > +| [`IMAGE_INSTALL`](#isar-image_install-variable) | [Root filesystem](#root-filesystem) | [Root filesystem](#root-filesystem) preparation and build | > > > > +| [`IMAGE_PREINSTALL`](#isar-image_preinstall-variable) | [Root filesystem](#root-filesystem) | | > > > > +| [`DEBIAN_DEPENDS`](#isar-debian_depends-variable) | [Root filesystem](#root-filesystem) | | > > > > +| [`Depends` control-file](#debian-depends-control-file-value) | [Root filesystem](#root-filesystem) | | > > > > +| [`IMAGE_TRANSIENT_PACKAGES`](#isar-image_transient_packages-variable) | [Root filesystem](#root-filesystem) | [Root filesystem](#root-filesystem) preparation and build | > > > > +| [`BUILDCHROOT_PREINSTALL`](#isar-buildchroot_preinstall-variable) | [Buildchroot](#buildroot) | | > > > > +| [`Build-Depends` control-file](#debian-build-depends-control-file-value) | [Buildchroot](#buildroot) | | > > > > +| [`IMAGER_BUILD_DEPS`](#isar-imager_build_deps-variable) | | [Imager](#imager) preparation | > > > > +| [`IMAGER_INSTALL`](#isar-imager_install-variable) | [Imager](#imager) | | > > > > +| [`WIC_IMAGER_INSTALL`](#isar-wic_imager_install-variable) | [Imager](#imager) | | > > > > + > > > > +## Dependencies specification variables > > > > + > > > > +### Bitbake `DEPENDS` variable > > > > + > > > > +This is a **Bitbake** variable that specifies **Bitbake inter-task dependencies**. > > > > + > > > > +Since it specifies the recipes that another recipe depend upon, this variable can be specified in **package recipes** and **image recipes**. > > > > + > > > > +### ISAR `IMAGE_INSTALL` variable > > > > + > > > > +This is originally a **Bitbake** variable that has conceptually the same goal in ISAR as in Yocto. > > > > +In the case of ISAR it specifies at the same time **Bitbake inter-task dependencies** and **image custom Debian package dependencies**. > > > > + > > > > +It basically specifies that a **[root filesystem](#root-filesystem)** (the prefix 'IMAGE' can be misleading here) requires a custom Debian package. > > > > +But since custom Debian packages are being built from package recipes, this variable also specifies that an image recipe depends on package recipes. > > > > + > > > > +For this "magic" to work both the Debian package and the package recipe have to have the same name. > > > > +Something else than this 1:1 behaviour (e.g. a recipe creating multiple Debian binary packages out of a single Debian source package) requires some hacking out of the scope of this document. > > > > + > > > > +This variable can be found in **configuration files** and **image recipes**. > > > > + > > > > +### ISAR `IMAGE_PREINSTALL` variable > > > > + > > > > +This is an **ISAR-only** variable that specifies **image upstream Debian package dependencies**. > > > > + > > > > +It specifies that a **[root filesystem](#root-filesystem)** requires certain *upstream Debian packages*, being "upstream Debian packages" ready built packages that can be downloaded from a > > > > Debian > > > > package repository. > > > > + > > > > +This variable can be found in **configuration files** and **image recipes**. > > > > + > > > > +### ISAR `DEBIAN_DEPENDS` variable > > > > + > > > > +This is an **ISAR-only** variable that specifies **Debian inter-package dependencies** and can be used only on package recipes inheriting from the **`dpkg-raw` class**. > > > > + > > > > +It lists the Debian binary packages that a custom Debian package depends upon. > > > > +It can be used for both upstream and custom packages. > > > > +It results in an entry in the corresponding **[Debian `Depends` control-file value](#debian-depends-control-file-value)**. > > > > + > > > > +A common usage is for creating meta-packages that only keep dependencies on further packages synchronized. > > > > + > > > > +This variable can be found in **package recipes**. > > > > + > > > > +### ISAR `IMAGE_TRANSIENT_PACKAGES` variable > > > > + > > > > +This is an **ISAR-only** variable that specifies **image custom Debian package dependencies** and **Bitbake inter-task dependencies**. > > > > + > > > > +This is a variable that is rarely needed and only for a very specific use case! > > > > +That is when a package is needed in the **[root filesystem](#root-filesystem)** during the **root filesystem build stage**, but not in the final **[root filesystem](#root-filesystem)**. > > > > +Meaning that the installation is only transient, therefore its name. > > > > + > > > > +This variable can be found in **configuration files** and **image recipes**. > > > > + > > > > +### Debian `Depends` control-file value > > > > + > > > > +This is an **Debian** variable that specifies **Debian inter-package dependencies**. > > > > + > > > > +`Depends` specifies the Debian packages that are required in the [runtime environment](#runtime-environment-specification) by the package declaring the dependencies. > > > > + > > > > +It relies on standard Debian mechanisms, therefore the corresponding [official Debian documentation][pkg-rels] provides the best reference. > > > > + > > > > +### ISAR `BUILDCHROOT_PREINSTALL` variable > > > > + > > > > +This is an **ISAR-only** variable that specifies **buildchroot Debian package dependencies**. > > > > + > > > > +It specifies that **[buildchroot](#buildroot)** requires certain *Debian packages*. > > > > + > > > > +This variable can be found in **buildchroot recipes**. > > > > + > > > > +### Debian `Build-Depends` control-file value > > > > + > > > > +This is an **Debian** variable that specifies **Debian inter-package dependencies**. > > > > + > > > > +`Build-Depends` specifies the Debian packages that are required in the [buildtime environment](#buildtime-environment-specification) to build the package declaring the dependencies. > > > > + > > > > +It relies on standard Debian mechanisms, therefore the corresponding [official Debian documentation][pkg-rels] provides the best reference. > > > > + > > > > +### ISAR `IMAGER_BUILD_DEPS` variable > > > > + > > > > +This is an **ISAR-only** variable that specifies **Bitbake inter-task dependencies**. > > > > + > > > > +It has exactly the same effect as `DEPENDS`. > > > > +It simply provides syntactic sugar to separate dependencies on recipes for building the runtime environment from dependencies on recipes for building the **[imager](#imager)**. > > > > + > > > > +This variable can be found in **configuration files**. > > > > + > > > > +### ISAR `IMAGER_INSTALL` variable > > > > + > > > > +This is an **ISAR-only** variable that specifies **imager Debian package dependencies**. > > > > + > > > > +It specifies that certain *Debian packages* have to be installed on the **[imager](#imager)**. > > > > + > > > > +This variable can be found typically in **configuration files**, although it could also be part of **image recipes**. > > > > + > > > > +### ISAR `WIC_IMAGER_INSTALL` variable > > > > + > > > > +This is an **ISAR-only** variable that specifies **imager Debian package dependencies**. > > > > + > > > > +It specifies that an **[imager](#imager)** requires certain *Debian packages* so that WIC can build the disk images (e.g. `parted`). > > > > + > > > > +This variable can be found typically in **configuration files**, although it could also be part of **image recipes**. > > > > + > > > > +[wks]: https://www.yoctoproject.org/docs/current/ref-manual/ref-manual.html#ref-kickstart "OpenEmbedded Kickstart (.wks) Reference" > > > > +[pkg-rels]: https://www.debian.org/doc/debian-policy/ch-relationships.html "Debian policy: Declaring relationships between packages" > > > > -- > > > > 2.11.0 > > > > > > > > > > > > > -- > > > Maxim Osipov > > > ilbers GmbH > > > Maria-Merian-Str. 8 > > > 85521 Ottobrunn > > > Germany > > > +49 (151) 6517 6917 > > > mosipov@ilbers.de > > > http://ilbers.de/ > > > Commercial register Munich, HRB 214197 > > > General Manager: Baurzhan Ismagulov > > > > > -- With best regards, Silvano Cirujano Cuesta Siemens AG Smart Embedded Systems Mobile: +49 173 3181462 mailto:silvano.cirujano-cuesta@siemens.com ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2019-04-03 9:06 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-02-21 15:33 [RFC] v2 Documentation: dependencies specification in Bitbake and Debian Cirujano Cuesta, Silvano 2019-03-28 18:30 ` Henning Schild 2019-03-29 13:02 ` Cirujano Cuesta, Silvano 2019-03-29 13:17 ` Cirujano Cuesta, Silvano 2019-04-03 7:19 ` Maxim Yu. Osipov 2019-04-03 8:24 ` Cirujano Cuesta, Silvano 2019-04-03 8:32 ` Maxim Yu. Osipov 2019-04-03 9:06 ` Cirujano Cuesta, Silvano
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox