* [RFC PATCH v3 0/1] Documentation: dependencies specification in Bitbake and Debian @ 2019-04-03 9:13 Silvano Cirujano Cuesta 2019-04-03 9:13 ` [RFC PATCH v3 1/1] docs: ading dependencies documentation Silvano Cirujano Cuesta 0 siblings, 1 reply; 3+ messages in thread From: Silvano Cirujano Cuesta @ 2019-04-03 9:13 UTC (permalink / raw) To: isar-users Hi, I've tried to document my understanding on dependency management at different areas in ISAR. I've done it mostly to structure it in my head. But I'm also convinced that this kind of documentation is critical for lowering the entry barrier to ISAR usage, therefore I'm sending it to this mailing list. With this request for comments I want to: 1. gather comments about the demand for such documentation 2. gather comments about the content to make sure that I didn't misunderstood how it works Once I get confirmation for the demand and any possible misconception clarified, I'll provide a patch integrating it with the already existing documentation. I've taken Claudius feedback to previous versions of the RFC into account (thank you again for your very valuable input!). And I let two colleagues of mine with a similar superficial/deep knowledge of ISAR review it. Right now it's everything in a single MarkDown document, but I'd rather put the section "Dependency specification variables" into a separate document. I keep it together right now to simplify the reviews. For those of you who like to have some comfort in life, I've pushed the document to a fork of ISAR in my personal area. That way you can take advantage of MarkDown conversion to HTML and rendering going to [1] ;-) Regards, Silvano [1] https://github.com/Silvanoc/isar/blob/silvano/dependencies-documentation/doc/dependencies.md Silvano Cirujano Cuesta (1): docs: ading dependencies documentation doc/dependencies.md | 255 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 255 insertions(+) create mode 100644 doc/dependencies.md -- 2.11.0 ^ permalink raw reply [flat|nested] 3+ messages in thread
* [RFC PATCH v3 1/1] docs: ading dependencies documentation 2019-04-03 9:13 [RFC PATCH v3 0/1] Documentation: dependencies specification in Bitbake and Debian Silvano Cirujano Cuesta @ 2019-04-03 9:13 ` Silvano Cirujano Cuesta 2019-04-08 10:44 ` Andreas Reichel 0 siblings, 1 reply; 3+ messages in thread From: Silvano Cirujano Cuesta @ 2019-04-03 9:13 UTC (permalink / raw) To: isar-users Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com> --- doc/dependencies.md | 255 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 255 insertions(+) create mode 100644 doc/dependencies.md diff --git a/doc/dependencies.md b/doc/dependencies.md new file mode 100644 index 0000000..390e677 --- /dev/null +++ b/doc/dependencies.md @@ -0,0 +1,255 @@ +# Dependency management in ISAR + +## Introduction + +The typical goal of an ISAR build is to obtain: + +1. a disk image containing one or more partition images (all optional), +1. containing one or more Debian root filesystems with +1. certain Debian packages installed. + +Reaching this goal relies on the availability of the above listed elements on *runtime*, the specification of the **[runtime environments](#runtime-environment-specification)** helps the different build tools ensuring the availability of the mentioned elements on runtime. + +Usually some of the runtime dependencies have to be built, and it happens in special environments. These **[buildtime environments](#buildtime-environment-specification)** help the different build tools preparing the environment for building the runtime dependencies. + +The differentiation between buildtime and runtime environments is very important. +The fact that ISAR uses Debian both for buildtime and runtime environments might mislead some people. +In general the names of the variables being used to specify those dependencies aren't supporting this differentiation, but difficulting it. + +Not only the build environments have to be specified, but also the **[build process](#build-process-specification)** that takes care of building and integrating both the runtime dependencies and the buildtime dependencies. + +## Runtime environment specification + +As mentioned above, ISAR can build [partitioned disk images](#disk-image) or only [root filesystems](#root-filesystem). + +Let's first analyze how the content of the different runtime containers can be specified from bigger to smaller container. + +``` ++---------------------------------------------------------------------------------------------------+ +| | +| DISK IMAGE | +| | +| +------------+ +-----------------------------------+ +------------------------------+ | +| | | | | | || | +| | Bootloader | | Bootable partition | | Other partitions ||| | +| | | | | | |||| | +| | | | +------------------------------+ | | |||| | +| | | | | | | | |||| | +| | | | | ROOT FILESYSTEM | | | |||| | +| +------------+ | | | | | |||| | +| | | +-------------------+ | | | |||| | +| | | | | | | | | |||| | +| | | | DEBIAN PACKAGES | | | | | |||| | +| | | | | | | | | |||| | +| | | | | | | | | |||| | +| | | | | | | | | |||| | +| | | +-------------------+ | | | | |||| | +| | | ---------------------+ | | | |||| | +| | | | | +------------------------------+||| | +| | +------------------------------+ | -------------------------------+|| | +| | | --------------------------------+| | +| +-----------------------------------+ --------------------------------+ | +| | ++---------------------------------------------------------------------------------------------------+ +``` + +### Disk image + +A disk image depends on: + +1. a bootloader (optional) and +1. one or more partitions, containing one of them a [root filesystem](#root-filesystem). + +The ISAR way to **specify** them is through *WKS files* (OpenEmbedded KickStart files). +ISAR doesn't change it in any significant way, therefore the corresponding [Yocto documentation][wks] provides the best reference. + +### Root filesystem + +A root filesystem depends on: + +1. a minimal set of files (the result of `debootstrap`) and +1. additional files and configurations obtained from *Debian packages*. + +The ISAR way to **specify** them is quite complex and involves *ISAR variables* that can be found in configuration files or recipes and artifacts and metadata being provided by *Debian packages*. + +See following sections for more information about how to specify the content of the root filesystem: + +* [ISAR `IMAGE_INSTALL` variable](#isar-image_install-variable) +* [ISAR `IMAGE_PREINSTALL` variable](#isar-image_preinstall-variable) +* [ISAR `DEBIAN_DEPENDS` variable](#isar-debian_depends-variable) +* [Debian `Depends` control-file value](#debian-depends-control-file-value) + +## Buildtime environment specification + +The creation of the [above mentioned runtime units](#runtime-environment-specification) (disk images, bootloaders, root filesystems, packages, ...) takes place in different environments. + +These environments have to be specified. + +### Buildchroot + +The *Debian packages* are built in the so-called *buildchroot* filesystem. + +It's a Debian chroot that is used to build custom Debian packages required by the **[root filesystem](#root-filesystem)**. +This build environment depends on the presence of certain artifacts (compilers, configurations, linkers, ...). + +The ISAR way to **specify** them is quite complex and involves *ISAR variables* that can be found in configuration files or recipes and artifacts and metadata being provided by *Debian packages*. + +See following sections for more information about how to specify the content of the chroot: + +* [ISAR `IMAGER_INSTALL` variable](#isar-image_install-variable) +* [ISAR `BUILDCHROOT_PREINSTALL` variable](#isar-buildchroot_preinstall-variable) +* [Debian `Build-Depends` control-file value](#debian-build-depends-control-file-value) + +If a disk image is being created, the same environment is being used for building the image. +For this usage additional dependencies might appear. See the [imager](#imager) section for more information. + +### Imager + +The *disk images* are built in the so-called *imager*. + +It's a Debian chroot that is used to build **[disk images](#disk-image)**. +This build environment depends on the presence of certain artifacts (compilers, configurations, linkers, ...). +Although conceptually the imager doesn't have anything to do with the buildchroot, in the implementation the same chroot is being used for both tasks. + +The ISAR way to **specify** them involves *ISAR variables* that can be found in configuration files or recipes. + +See following sections for more information about how to specify the content of the chroot: + +* [ISAR `WIC_IMAGER_INSTALL` variable](#isar-wic_imager_install-variable) +* [ISAR `IMAGER_BUILD_DEPS` variable](#isar-imager_build_deps-variable) + +### Target filesystem + +It's a Debian root filesystem that results from creating a minimalistic Debian skeleton via `debootstrap` and installing Debian packages on it, resulting in the [root filesystem](#root-filesystem). + +There's no need to **specify** this build environment, since it's the [root filesystem of the runtime environment](#root-filesystem). + +## Build process specification + +ISAR relies on a subset of Bitbake to take care of the build process. +Therefore the ISAR way to **specify** the different tasks involved in the process is Bitbake recipes with some *Bitbake variables* (well documented ) and some *ISAR variables* (some of them are only syntactic sugar). + +See following sections for more information about how to specify inter-tasks dependencies: + +* [Bitbake `DEPENDS` variable](#bitbake-depends-variable) +* [ISAR `IMAGE_INSTALL` variable](#isar-image_install-variable) +* [ISAR `IMAGER_BUILD_DEPS` variable](#isar-imager_build_deps-variable) + +## Mapping: Variables <=> Build environment and Stages + +| Variable | Environment | Stage | +|:-------- |:-----------------:|:-----:| +| [`DEPENDS`](#bitbake-depends-variable) | | All | +| [`IMAGE_INSTALL`](#isar-image_install-variable) | [Root filesystem](#root-filesystem) | [Root filesystem](#root-filesystem) preparation and build | +| [`IMAGE_PREINSTALL`](#isar-image_preinstall-variable) | [Root filesystem](#root-filesystem) | | +| [`DEBIAN_DEPENDS`](#isar-debian_depends-variable) | [Root filesystem](#root-filesystem) | | +| [`Depends` control-file](#debian-depends-control-file-value) | [Root filesystem](#root-filesystem) | | +| [`IMAGE_TRANSIENT_PACKAGES`](#isar-image_transient_packages-variable) | [Root filesystem](#root-filesystem) | [Root filesystem](#root-filesystem) preparation and build | +| [`BUILDCHROOT_PREINSTALL`](#isar-buildchroot_preinstall-variable) | [Buildchroot](#buildroot) | | +| [`Build-Depends` control-file](#debian-build-depends-control-file-value) | [Buildchroot](#buildroot) | | +| [`IMAGER_BUILD_DEPS`](#isar-imager_build_deps-variable) | | [Imager](#imager) preparation | +| [`IMAGER_INSTALL`](#isar-imager_install-variable) | [Imager](#imager) | | +| [`WIC_IMAGER_INSTALL`](#isar-wic_imager_install-variable) | [Imager](#imager) | | + +## Dependencies specification variables + +### Bitbake `DEPENDS` variable + +This is a **Bitbake** variable that specifies **Bitbake inter-task dependencies**. + +Since it specifies the recipes that another recipe depend upon, this variable can be specified in **package recipes** and **image recipes**. + +### ISAR `IMAGE_INSTALL` variable + +This is originally a **Bitbake** variable that has conceptually the same goal in ISAR as in Yocto. +In the case of ISAR it specifies at the same time **Bitbake inter-task dependencies** and **image custom Debian package dependencies**. + +It basically specifies that a **[root filesystem](#root-filesystem)** (the prefix 'IMAGE' can be misleading here) requires a custom Debian package. +But since custom Debian packages are being built from package recipes, this variable also specifies that an image recipe depends on package recipes. + +For this "magic" to work both the Debian package and the package recipe have to have the same name. +Something else than this 1:1 behaviour (e.g. a recipe creating multiple Debian binary packages out of a single Debian source package) requires some hacking out of the scope of this document. + +This variable can be found in **configuration files** and **image recipes**. + +### ISAR `IMAGE_PREINSTALL` variable + +This is an **ISAR-only** variable that specifies **image upstream Debian package dependencies**. + +It specifies that a **[root filesystem](#root-filesystem)** requires certain *upstream Debian packages*, being "upstream Debian packages" ready built packages that can be downloaded from a Debian package repository. + +This variable can be found in **configuration files** and **image recipes**. + +### ISAR `DEBIAN_DEPENDS` variable + +This is an **ISAR-only** variable that specifies **Debian inter-package dependencies** and can be used only on package recipes inheriting from the **`dpkg-raw` class**. + +It lists the Debian binary packages that a custom Debian package depends upon. +It can be used for both upstream and custom packages. +It results in an entry in the corresponding **[Debian `Depends` control-file value](#debian-depends-control-file-value)**. + +A common usage is for creating meta-packages that only keep dependencies on further packages synchronized. + +This variable can be found in **package recipes**. + +### ISAR `IMAGE_TRANSIENT_PACKAGES` variable + +This is an **ISAR-only** variable that specifies **image custom Debian package dependencies** and **Bitbake inter-task dependencies**. + +This is a variable that is rarely needed and only for a very specific use case! +That is when a package is needed in the **[root filesystem](#root-filesystem)** during the **root filesystem build stage**, but not in the final **[root filesystem](#root-filesystem)**. +Meaning that the installation is only transient, therefore its name. + +This variable can be found in **configuration files** and **image recipes**. + +### Debian `Depends` control-file value + +This is an **Debian** variable that specifies **Debian inter-package dependencies**. + +`Depends` specifies the Debian packages that are required in the [runtime environment](#runtime-environment-specification) by the package declaring the dependencies. + +It relies on standard Debian mechanisms, therefore the corresponding [official Debian documentation][pkg-rels] provides the best reference. + +### ISAR `BUILDCHROOT_PREINSTALL` variable + +This is an **ISAR-only** variable that specifies **buildchroot Debian package dependencies**. + +It specifies that **[buildchroot](#buildroot)** requires certain *Debian packages*. + +This variable can be found in **buildchroot recipes**. + +### Debian `Build-Depends` control-file value + +This is an **Debian** variable that specifies **Debian inter-package dependencies**. + +`Build-Depends` specifies the Debian packages that are required in the [buildtime environment](#buildtime-environment-specification) to build the package declaring the dependencies. + +It relies on standard Debian mechanisms, therefore the corresponding [official Debian documentation][pkg-rels] provides the best reference. + +### ISAR `IMAGER_BUILD_DEPS` variable + +This is an **ISAR-only** variable that specifies **Bitbake inter-task dependencies**. + +It has exactly the same effect as `DEPENDS`. +It simply provides syntactic sugar to separate dependencies on recipes for building the runtime environment from dependencies on recipes for building the **[imager](#imager)**. + +This variable can be found in **configuration files**. + +### ISAR `IMAGER_INSTALL` variable + +This is an **ISAR-only** variable that specifies **imager Debian package dependencies**. + +It specifies that certain *Debian packages* have to be installed on the **[imager](#imager)**. + +This variable can be found typically in **configuration files**, although it could also be part of **image recipes**. + +### ISAR `WIC_IMAGER_INSTALL` variable + +This is an **ISAR-only** variable that specifies **imager Debian package dependencies**. + +It specifies that an **[imager](#imager)** requires certain *Debian packages* so that WIC can build the disk images (e.g. `parted`). + +This variable can be found typically in **configuration files**, although it could also be part of **image recipes**. + +[wks]: https://www.yoctoproject.org/docs/current/ref-manual/ref-manual.html#ref-kickstart "OpenEmbedded Kickstart (.wks) Reference" +[pkg-rels]: https://www.debian.org/doc/debian-policy/ch-relationships.html "Debian policy: Declaring relationships between packages" -- 2.11.0 ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [RFC PATCH v3 1/1] docs: ading dependencies documentation 2019-04-03 9:13 ` [RFC PATCH v3 1/1] docs: ading dependencies documentation Silvano Cirujano Cuesta @ 2019-04-08 10:44 ` Andreas Reichel 0 siblings, 0 replies; 3+ messages in thread From: Andreas Reichel @ 2019-04-08 10:44 UTC (permalink / raw) To: [ext] Silvano Cirujano Cuesta; +Cc: isar-users On Wed, Apr 03, 2019 at 11:13:04AM +0200, [ext] Silvano Cirujano Cuesta wrote: > Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com> > --- > doc/dependencies.md | 255 ++++++++++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 255 insertions(+) > create mode 100644 doc/dependencies.md > > diff --git a/doc/dependencies.md b/doc/dependencies.md > new file mode 100644 > index 0000000..390e677 > --- /dev/null > +++ b/doc/dependencies.md > @@ -0,0 +1,255 @@ > +# Dependency management in ISAR > + > +## Introduction > + > +The typical goal of an ISAR build is to obtain: > + > +1. a disk image containing one or more partition images (all optional), > +1. containing one or more Debian root filesystems with > +1. certain Debian packages installed. > (Should be 1., 2. and 3., so that one can easier read it as normal text as well) > +Reaching this goal relies on the availability of the above listed elements on *runtime*, the specification of the **[runtime environments](#runtime-environment-specification)** helps the different build tools ensuring the availability of the mentioned elements on runtime. > + > +Usually some of the runtime dependencies have to be built, and it happens in special environments. These **[buildtime environments](#buildtime-environment-specification)** help the different build tools preparing the environment for building the runtime dependencies. > + > +The differentiation between buildtime and runtime environments is very important. > +The fact that ISAR uses Debian both for buildtime and runtime environments might mislead some people. > +In general the names of the variables being used to specify those dependencies aren't supporting this differentiation, but difficulting it. There is no 'difficulting'. => but making it difficult. / but encumbering it. (Is it a good idea to document, that ISAR makes it harder to people instead of easier? :D) > + > +Not only the build environments have to be specified, but also the **[build process](#build-process-specification)** that takes care of building and integrating both the runtime dependencies and the buildtime dependencies. > + > +## Runtime environment specification > + > +As mentioned above, ISAR can build [partitioned disk images](#disk-image) or only [root filesystems](#root-filesystem). > + > +Let's first analyze how the content of the different runtime containers can be specified from bigger to smaller container. ... from bigger to smaller containerS > + > + > +### Disk image > + > +A disk image depends on: > + > +1. a bootloader (optional) and > +1. one or more partitions, containing one of them a [root filesystem](#root-filesystem). > + , one of them containing a [root filesystem] > + > +See following sections for more information about how to specify inter-tasks dependencies: intertask dependencies > + > + > +## Dependencies specification variables ## Variables for dependency specification > + > +### Bitbake `DEPENDS` variable > + > +This is a **Bitbake** variable that specifies **Bitbake inter-task dependencies**. > + intertask dependencies > +Since it specifies the recipes that another recipe depend upon, this variable can be specified in **package recipes** and **image recipes**. > + > +### ISAR `IMAGE_INSTALL` variable > + > +This is originally a **Bitbake** variable that has conceptually the same goal in ISAR as in Yocto. that conceptually has > +In the case of ISAR it specifies at the same time **Bitbake inter-task dependencies** and **image custom Debian package dependencies**. > + In the case of ISAR it specifies **Bitbake intertask dependencies** and **image custom Debian package dependencies** at the same time. > +It basically specifies that a **[root filesystem](#root-filesystem)** (the prefix 'IMAGE' can be misleading here) requires a custom Debian package. > +But since custom Debian packages are being built from package recipes, this variable also specifies that an image recipe depends on package recipes. > + > +For this "magic" to work both the Debian package and the package recipe have to have the same name. , > +Something else than this 1:1 behaviour (e.g. a recipe creating multiple Debian binary packages out of a single Debian source package) requires some hacking out of the scope of this document. > + hacking, which is out of the scope of this document. > +This variable can be found in **configuration files** and **image recipes**. > + > +### ISAR `IMAGE_PREINSTALL` variable > + > +This is an **ISAR-only** variable that specifies **image upstream Debian package dependencies**. > + > +It specifies that a **[root filesystem](#root-filesystem)** requires certain *upstream Debian packages*, being "upstream Debian packages" ready built packages that can be downloaded from a Debian package repository. > + being "upstream Debian packages" ready built packages.... .???????? I don't understand that :) > +This is a variable that is rarely needed and only for a very specific use case! > +That is when a package is needed in the **[root filesystem](#root-filesystem)** during the **root filesystem build stage**, but not in the final **[root filesystem](#root-filesystem)**. > +Meaning that the installation is only transient, therefore its name. As the name says, the installation is only transient. > + > +This variable can be found in **configuration files** and **image recipes**. > + > +### Debian `Depends` control-file value > + > +This is an **Debian** variable that specifies **Debian inter-package dependencies**. > + This is a > +### Debian `Build-Depends` control-file value > + > +This is an **Debian** variable that specifies **Debian inter-package dependencies**. This is a > + > +`Build-Depends` specifies the Debian packages that are required in the [buildtime environment](#buildtime-environment-specification) to build the package declaring the dependencies. > + > +It relies on standard Debian mechanisms, therefore the corresponding [official Debian documentation][pkg-rels] provides the best reference. > + > +### ISAR `IMAGER_BUILD_DEPS` variable > + > +This is an **ISAR-only** variable that specifies **Bitbake inter-task dependencies**. > + intertask > +It has exactly the same effect as `DEPENDS`. > +It simply provides syntactic sugar to separate dependencies on recipes for building the runtime environment from dependencies on recipes for building the **[imager](#imager)**. It simply provides syntactic sugar to separate dependencies on recipes for building the runtime environment from those for building the **[imager](#imager)**. > + > +This variable can be found in **configuration files**. > + > +### ISAR `IMAGER_INSTALL` variable > + > +This is an **ISAR-only** variable that specifies **imager Debian package dependencies**. > + > +It specifies that certain *Debian packages* have to be installed on the **[imager](#imager)**. installed FOR the > + > +This variable can be found typically in **configuration files**, although it could also be part of **image recipes**. > + can typically be found can also be used in **image recipes**. > +### ISAR `WIC_IMAGER_INSTALL` variable > + > +This is an **ISAR-only** variable that specifies **imager Debian package dependencies**. > + > +It specifies that an **[imager](#imager)** requires certain *Debian packages* so that WIC can build the disk images (e.g. `parted`). > + > +This variable can be found typically in **configuration files**, although it could also be part of **image recipes**. > + can typically be found in can also be used in **image recipes**. > +[wks]: https://www.yoctoproject.org/docs/current/ref-manual/ref-manual.html#ref-kickstart "OpenEmbedded Kickstart (.wks) Reference" > +[pkg-rels]: https://www.debian.org/doc/debian-policy/ch-relationships.html "Debian policy: Declaring relationships between packages" > -- > 2.11.0 KR Andreas > > -- > You received this message because you are subscribed to the Google Groups "isar-users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. > To post to this group, send email to isar-users@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20190403091304.8087-2-silvano.cirujano-cuesta%40siemens.com. > For more options, visit https://groups.google.com/d/optout. -- Andreas Reichel Dipl.-Phys. (Univ.) Software Consultant Andreas.Reichel@tngtech.com, +49-174-3180074 TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082 ^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2019-04-08 10:44 UTC | newest] Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-04-03 9:13 [RFC PATCH v3 0/1] Documentation: dependencies specification in Bitbake and Debian Silvano Cirujano Cuesta 2019-04-03 9:13 ` [RFC PATCH v3 1/1] docs: ading dependencies documentation Silvano Cirujano Cuesta 2019-04-08 10:44 ` Andreas Reichel
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox