[RFC] Documentation: dependencies specification in Bitbake and Debian

["Cirujano Cuesta, Silvano" <silvano.cirujano-cuesta@siemens.com>]

Hi,

I've tried to document my understanding on dependency management at different areas in ISAR.
I've done it mostly to structure it in my head.
But I'm also convinced that this kind of documentation is critical for lowering the entry barrier to ISAR usage, therefore I'm sending it to this mailing list.

With this request for comments I want to:
1. gather comments about the demand for such documentation
2. gather comments about the content to make sure that I didn't misunderstood how it works

Once I get confirmation for the demand and any possible misconception clarified, I'll provide a patch integrating it with the already existing documentation.

Please keep comments about the container in the chamber. You can shoot them later when I send the patch ;-)

You can find the content in Markdown format at the bottom.

Regards,
   Silvano

--------------------------------

# Dependency management in ISAR

## Dependencies resolution

Dependencies have to be taken into account separately on different stages:

1. Builtime dependencies
   1. [Bitbake buildtime dependencies](#bitbake-buildtime-dependencies)
   1. [Debian buildtime dependencies](#debian-buildtime-dependencies)
1. Runtime dependencies
   1. [Functional runtime dependencies](#functional-runtime-dependencies)
   1. [Package runtime dependencies](#package-runtime-dependencies)

### Bitbake buildtime dependencies

The execution of a Bitbake recipe might require the execution of other recipes, these are the Bitbake buildtime dependencies.

### Debian buildtime dependencies

Debian is not only the distribution of the resulting root filesystem, but also the distribution of the build environment.
Debian buildtime dependencies are Debian packages needed in the build environment to build certain Debian package.

Following sections show that ISAR doesn't provide any automatic mechanism to specify these dependencies.
Therefore [`Build-Depends`][pkg-rels] has to be specified in the corresponding Debian Control file (`debian/control`).

### Functional runtime dependencies

These are Debian packages needed to provide certain functionality in the target.

They are explicitly installed in the image, therefore will be considered as "manually" installed by APT, as a consequence they can only be removed manually.

Functional runtime dependencies can be seen as image dependencies, since they are required for the image to provide the expected functionality.

### Package runtime dependencies

These are Debian packages needed by packages being "manually" installed. They are not needed by themselves, but only to obtain the functionality provided by the [functional runtime
dependencies](#functional-runtime-dependencies)

Upstream Debian packages already provide their own package runtime dependencies.

Custom packages have to provide these dependencies. They can be specified via the corresponding Debian Control files (`debian/control`).

ISAR supports on specifying these dependencies if inheriting from the [`dpkg-raw` class][dpkg-raw].

These packages will be considered as "automatically" installed by APT as dependencies of other packages, as a consequence APT can identify when they are not needed anymore.
APT provides the subcommand `autoremove` to uninstall them.

## Dependencies table

Acronyms:
> Bt = Buildtime
>
> Rt = Runtime

| Variable           | Bt Bitbake | Bt Debian | Rt Functional | Rt Package |
| ------------------ |:----------:|:---------:|:-------------:|:----------:|
| `DEPENDS`          |     x      |           |               |            |
| `IMAGE_INSTALL`    |     x      |           |      x        |            |
| `IMAGE_PREINSTALL` |            |           |      x        |            |
| `DEBIAN_DEPENDS`   |            |           |               |     x      |

## How to control buildtime dependencies

As mentioned above, buildtime dependencies specify which Bitbake recipes get executed and Debian packages get built.

The union of recipes specified via `DEPENDS` and `IMAGE_INSTALL` and after applying filters like `PREFERRED_VERSION` gets selected to be [executed][bb-tasks].

### Bitbake buildtime: `DEPENDS`

`DEPENDS` specifies recipes available in one of the layers that have to be built.

The logical place to use this variable depends on which kind of dependency it should specify.

Is it a recipe that builds a package that requires another custom package (built from a Bitbake recipe)?
Then probably a **package** recipe is the best place.
The [above table](#dependencies-table) shows that there is no way to specify at the same time a Bitbake runtime dependency and a Debian Package dependency, therefore the corresponding [package runtime
dependency](#package-runtime-dependencies) should be also added.

Otherwise probably an **image** recipe or **configuration** file is the best place.

Opposed to `IMAGE_INSTALL`, this doesn't affect any runtime dependencies.

This variable can be seen as the counterpart at buildtime for [`IMAGE_PREINSTALL`](#functional-runtime-image_preinstall) (runtime) and [`DEBIAN_DEPENDS`](#package-runtime-debian_depends) (runtime).

### Bitbake buildtime: `IMAGE_INSTALL`

`IMAGE_INSTALL` specifies recipes available in one of the layers that have to be built.
And at the same time specifies that the Debian packages resulting from the recipes have to be installed in the image (see [below](#functional-runtime-image_install) for more information).

The logical place to use this variable is an **image** recipe or **configuration** file (distro, layer, ...).

## How to control runtime dependencies

Runtime dependencies specify which Debian packages are made availabe during runtime (by installing them in the image, if one is being created).

The union of packages specified via `IMAGE_INSTALL`, `IMAGE_PREINSTALL`, `DEBIAN_DEPENDS`, `debian/control` files of custom packages and dependencies of upstream Debian packages get installed into the
image.

### Functional runtime: `IMAGE_INSTALL`

`IMAGE_INSTALL` specifies custom Debian packages that have to be installed in the image.
And at the same time specifies that the correponding Bitbake recipes have to be previously executed to build the Debian packages (see [above](#bitbake-buildtime-image_install) for more information).

The logical place to use this variable is an **image** recipe or **configuration** file (distro, layer, ...).

### Functional runtime: `IMAGE_PREINSTALL`

`IMAGE_PREINSTALL` specifies upstream Debian packages (opposed to custom Debian packages built from Bitbake recipes) that are needed in the image.

The logical place to use this variable is an **image** recipe or **configuration** file (distro, layer, ...).

These packages will be marked as "manually installed" (they can be shown with `apt-mark showmanual`).

Opposed to `IMAGE_INSTALL`, this doesn't affect any dependencies at buildtime.

This variable can be seen as one counterpart at runtime for [`DEPENDS`](#bitbake-buildtime-depends) at buildtime.

### Package runtime: `DEBIAN_DEPENDS`

`DEBIAN_DEPENDS` specifies the packages that are required by the package being built.
They will be added to the [`Depends` of the package control file][pkg-rels] and therefore have to be specified in the format expected by Debian.

The logical place to use this variable is a **package** recipe.

These packages will be marked as "automatically installed" (they can be shown with `apt-mark showauto`).

Opposed to `IMAGE_INSTALL`, this doesn't affect any dependencies at builtime.

This variable can be seen as one counterpart at runtime for [`DEPENDS`](#bitbake-buildtime-depends) at buildtime.

## Bitbake variables being ignored by ISAR

This section is documenting Bitbake variables being used for package definition and dependency specification in OpenEmbedded and Yocto, but being ignored by ISAR!

The ISAR ways to accomplish the same goal are being also documented here.

### `RDEPENDS`

This variable ["lists a package's runtime dependencies"][bb-rdepends].

In ISAR there are two ways to achieve the same:

1. Specify the dependencies via [`DEBIAN_DEPENDS`](#package-runtime-debian_depends) if inheriting from the [`dpkg-raw` class][dpkg-raw]].
1. Adding them to the [`Depends` of the package `debian/control`][pkg-rels] file.

### `RRECOMMENDS`

This variable provides ["a list of packages that extends the usability of a package"][bb-rrecommends].

In ISAR there is only one way to achieve the same:

1. Adding them to the [`Recommends` of the package `debian/control`][pkg-rels] file.

### `PACKAGES`

This variable provides ["the list of packages the recipe creates"][bb-packages].

In ISAR there is only one way to achieve the same:

1. Specifying multiple packages in the [`debian/control` file][pkg-control] being provided by a recipe.

[pkg-rels]: https://www.debian.org/doc/debian-policy/ch-relationships.html "Debian policy: Declaring relationships between packages"
[pkg-control]: https://www.debian.org/doc/debian-policy/ch-controlfields.html "Debian policy: Control files and their fields"
[dpkg-raw]: https://github.com/ilbers/isar/blob/6c5db020b9b837d7b0ce63bfc719f9192e725f26/meta/classes/dpkg-raw.bbclass
[bb-tasks]: https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html#executing-tasks "Bitbake: Executing Tasks"
[bb-rdepends]: https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html#var-RDEPENDS "Bitbake: RDEPENDS"
[bb-rrecommends]: https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html#var-RRECOMMENDS "Bitbake: RRECOMMENDS"
[bb-packages]: https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html#var-PACKAGES "Bitbake: PACKAGES"