[RFC] Documentation: dependencies specification in Bitbake and Debian

[RFC] Documentation: dependencies specification in Bitbake and Debian

[Cirujano Cuesta, Silvano]

Hi,

I've tried to document my understanding on dependency management at different areas in ISAR.
I've done it mostly to structure it in my head.
But I'm also convinced that this kind of documentation is critical for lowering the entry barrier to ISAR usage, therefore I'm sending it to this mailing list.

With this request for comments I want to:
1. gather comments about the demand for such documentation
2. gather comments about the content to make sure that I didn't misunderstood how it works

Once I get confirmation for the demand and any possible misconception clarified, I'll provide a patch integrating it with the already existing documentation.

Please keep comments about the container in the chamber. You can shoot them later when I send the patch ;-)

You can find the content in Markdown format at the bottom.

Regards,
   Silvano

--------------------------------

# Dependency management in ISAR

## Dependencies resolution

Dependencies have to be taken into account separately on different stages:

1. Builtime dependencies
   1. [Bitbake buildtime dependencies](#bitbake-buildtime-dependencies)
   1. [Debian buildtime dependencies](#debian-buildtime-dependencies)
1. Runtime dependencies
   1. [Functional runtime dependencies](#functional-runtime-dependencies)
   1. [Package runtime dependencies](#package-runtime-dependencies)

### Bitbake buildtime dependencies

The execution of a Bitbake recipe might require the execution of other recipes, these are the Bitbake buildtime dependencies.

### Debian buildtime dependencies

Debian is not only the distribution of the resulting root filesystem, but also the distribution of the build environment.
Debian buildtime dependencies are Debian packages needed in the build environment to build certain Debian package.

Following sections show that ISAR doesn't provide any automatic mechanism to specify these dependencies.
Therefore [`Build-Depends`][pkg-rels] has to be specified in the corresponding Debian Control file (`debian/control`).

### Functional runtime dependencies

These are Debian packages needed to provide certain functionality in the target.

They are explicitly installed in the image, therefore will be considered as "manually" installed by APT, as a consequence they can only be removed manually.

Functional runtime dependencies can be seen as image dependencies, since they are required for the image to provide the expected functionality.

### Package runtime dependencies

These are Debian packages needed by packages being "manually" installed. They are not needed by themselves, but only to obtain the functionality provided by the [functional runtime
dependencies](#functional-runtime-dependencies)

Upstream Debian packages already provide their own package runtime dependencies.

Custom packages have to provide these dependencies. They can be specified via the corresponding Debian Control files (`debian/control`).

ISAR supports on specifying these dependencies if inheriting from the [`dpkg-raw` class][dpkg-raw].

These packages will be considered as "automatically" installed by APT as dependencies of other packages, as a consequence APT can identify when they are not needed anymore.
APT provides the subcommand `autoremove` to uninstall them.

## Dependencies table

Acronyms:
> Bt = Buildtime
>
> Rt = Runtime

| Variable           | Bt Bitbake | Bt Debian | Rt Functional | Rt Package |
| ------------------ |:----------:|:---------:|:-------------:|:----------:|
| `DEPENDS`          |     x      |           |               |            |
| `IMAGE_INSTALL`    |     x      |           |      x        |            |
| `IMAGE_PREINSTALL` |            |           |      x        |            |
| `DEBIAN_DEPENDS`   |            |           |               |     x      |

## How to control buildtime dependencies

As mentioned above, buildtime dependencies specify which Bitbake recipes get executed and Debian packages get built.

The union of recipes specified via `DEPENDS` and `IMAGE_INSTALL` and after applying filters like `PREFERRED_VERSION` gets selected to be [executed][bb-tasks].

### Bitbake buildtime: `DEPENDS`

`DEPENDS` specifies recipes available in one of the layers that have to be built.

The logical place to use this variable depends on which kind of dependency it should specify.

Is it a recipe that builds a package that requires another custom package (built from a Bitbake recipe)?
Then probably a **package** recipe is the best place.
The [above table](#dependencies-table) shows that there is no way to specify at the same time a Bitbake runtime dependency and a Debian Package dependency, therefore the corresponding [package runtime
dependency](#package-runtime-dependencies) should be also added.

Otherwise probably an **image** recipe or **configuration** file is the best place.

Opposed to `IMAGE_INSTALL`, this doesn't affect any runtime dependencies.

This variable can be seen as the counterpart at buildtime for [`IMAGE_PREINSTALL`](#functional-runtime-image_preinstall) (runtime) and [`DEBIAN_DEPENDS`](#package-runtime-debian_depends) (runtime).

### Bitbake buildtime: `IMAGE_INSTALL`

`IMAGE_INSTALL` specifies recipes available in one of the layers that have to be built.
And at the same time specifies that the Debian packages resulting from the recipes have to be installed in the image (see [below](#functional-runtime-image_install) for more information).

The logical place to use this variable is an **image** recipe or **configuration** file (distro, layer, ...).

## How to control runtime dependencies

Runtime dependencies specify which Debian packages are made availabe during runtime (by installing them in the image, if one is being created).

The union of packages specified via `IMAGE_INSTALL`, `IMAGE_PREINSTALL`, `DEBIAN_DEPENDS`, `debian/control` files of custom packages and dependencies of upstream Debian packages get installed into the
image.

### Functional runtime: `IMAGE_INSTALL`

`IMAGE_INSTALL` specifies custom Debian packages that have to be installed in the image.
And at the same time specifies that the correponding Bitbake recipes have to be previously executed to build the Debian packages (see [above](#bitbake-buildtime-image_install) for more information).

The logical place to use this variable is an **image** recipe or **configuration** file (distro, layer, ...).

### Functional runtime: `IMAGE_PREINSTALL`

`IMAGE_PREINSTALL` specifies upstream Debian packages (opposed to custom Debian packages built from Bitbake recipes) that are needed in the image.

The logical place to use this variable is an **image** recipe or **configuration** file (distro, layer, ...).

These packages will be marked as "manually installed" (they can be shown with `apt-mark showmanual`).

Opposed to `IMAGE_INSTALL`, this doesn't affect any dependencies at buildtime.

This variable can be seen as one counterpart at runtime for [`DEPENDS`](#bitbake-buildtime-depends) at buildtime.

### Package runtime: `DEBIAN_DEPENDS`

`DEBIAN_DEPENDS` specifies the packages that are required by the package being built.
They will be added to the [`Depends` of the package control file][pkg-rels] and therefore have to be specified in the format expected by Debian.

The logical place to use this variable is a **package** recipe.

These packages will be marked as "automatically installed" (they can be shown with `apt-mark showauto`).

Opposed to `IMAGE_INSTALL`, this doesn't affect any dependencies at builtime.

This variable can be seen as one counterpart at runtime for [`DEPENDS`](#bitbake-buildtime-depends) at buildtime.

## Bitbake variables being ignored by ISAR

This section is documenting Bitbake variables being used for package definition and dependency specification in OpenEmbedded and Yocto, but being ignored by ISAR!

The ISAR ways to accomplish the same goal are being also documented here.

### `RDEPENDS`

This variable ["lists a package's runtime dependencies"][bb-rdepends].

In ISAR there are two ways to achieve the same:

1. Specify the dependencies via [`DEBIAN_DEPENDS`](#package-runtime-debian_depends) if inheriting from the [`dpkg-raw` class][dpkg-raw]].
1. Adding them to the [`Depends` of the package `debian/control`][pkg-rels] file.

### `RRECOMMENDS`

This variable provides ["a list of packages that extends the usability of a package"][bb-rrecommends].

In ISAR there is only one way to achieve the same:

1. Adding them to the [`Recommends` of the package `debian/control`][pkg-rels] file.

### `PACKAGES`

This variable provides ["the list of packages the recipe creates"][bb-packages].

In ISAR there is only one way to achieve the same:

1. Specifying multiple packages in the [`debian/control` file][pkg-control] being provided by a recipe.

[pkg-rels]: https://www.debian.org/doc/debian-policy/ch-relationships.html "Debian policy: Declaring relationships between packages"
[pkg-control]: https://www.debian.org/doc/debian-policy/ch-controlfields.html "Debian policy: Control files and their fields"
[dpkg-raw]: https://github.com/ilbers/isar/blob/6c5db020b9b837d7b0ce63bfc719f9192e725f26/meta/classes/dpkg-raw.bbclass
[bb-tasks]: https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html#executing-tasks "Bitbake: Executing Tasks"
[bb-rdepends]: https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html#var-RDEPENDS "Bitbake: RDEPENDS"
[bb-rrecommends]: https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html#var-RRECOMMENDS "Bitbake: RRECOMMENDS"
[bb-packages]: https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html#var-PACKAGES "Bitbake: PACKAGES"

Re: [RFC] Documentation: dependencies specification in Bitbake and Debian

[Claudius Heine]

Hi Silvano

On 06/02/2019 14.34, [ext] Cirujano Cuesta, Silvano wrote:
> Hi,
> 
> I've tried to document my understanding on dependency management at different areas in ISAR.
> I've done it mostly to structure it in my head.
> But I'm also convinced that this kind of documentation is critical for lowering the entry barrier to ISAR usage, therefore I'm sending it to this mailing list.
> 
> With this request for comments I want to:
> 1. gather comments about the demand for such documentation

Yes that would be good to have. IMO

> 2. gather comments about the content to make sure that I didn't misunderstood how it works

See below.

> 
> Once I get confirmation for the demand and any possible misconception clarified, I'll provide a patch integrating it with the already existing documentation.
> 
> Please keep comments about the container in the chamber. You can shoot them later when I send the patch ;-)
> 
> You can find the content in Markdown format at the bottom.
> 
> Regards,
>     Silvano
> 
> --------------------------------
> 
> # Dependency management in ISAR
> 
> ## Dependencies resolution
> 
> Dependencies have to be taken into account separately on different stages:
> 
> 1. Builtime dependencies
>     1. [Bitbake buildtime dependencies](#bitbake-buildtime-dependencies)
>     1. [Debian buildtime dependencies](#debian-buildtime-dependencies)
> 1. Runtime dependencies
>     1. [Functional runtime dependencies](#functional-runtime-dependencies)
>     1. [Package runtime dependencies](#package-runtime-dependencies)
> 
> ### Bitbake buildtime dependencies
> 
> The execution of a Bitbake recipe might require the execution of other recipes, these are the Bitbake buildtime dependencies.
> 
> ### Debian buildtime dependencies
> 
> Debian is not only the distribution of the resulting root filesystem, but also the distribution of the build environment.
> Debian buildtime dependencies are Debian packages needed in the build environment to build certain Debian package.
> 
> Following sections show that ISAR doesn't provide any automatic mechanism to specify these dependencies.
> Therefore [`Build-Depends`][pkg-rels] has to be specified in the corresponding Debian Control file (`debian/control`).
> 
> ### Functional runtime dependencies
> 
> These are Debian packages needed to provide certain functionality in the target.
> 
> They are explicitly installed in the image, therefore will be considered as "manually" installed by APT, as a consequence they can only be removed manually.
> 
> Functional runtime dependencies can be seen as image dependencies, since they are required for the image to provide the expected functionality.
> 
> ### Package runtime dependencies
> 
> These are Debian packages needed by packages being "manually" installed. They are not needed by themselves, but only to obtain the functionality provided by the [functional runtime
> dependencies](#functional-runtime-dependencies)
> 
> Upstream Debian packages already provide their own package runtime dependencies.
> 
> Custom packages have to provide these dependencies. They can be specified via the corresponding Debian Control files (`debian/control`).
> 
> ISAR supports on specifying these dependencies if inheriting from the [`dpkg-raw` class][dpkg-raw].
> 
> These packages will be considered as "automatically" installed by APT as dependencies of other packages, as a consequence APT can identify when they are not needed anymore.
> APT provides the subcommand `autoremove` to uninstall them.
> 
> ## Dependencies table
> 
> Acronyms:
>> Bt = Buildtime
>>
>> Rt = Runtime
> 
> | Variable           | Bt Bitbake | Bt Debian | Rt Functional | Rt Package |
> | ------------------ |:----------:|:---------:|:-------------:|:----------:|
> | `DEPENDS`          |     x      |           |               |            |
> | `IMAGE_INSTALL`    |     x      |           |      x        |            | > | `IMAGE_PREINSTALL` 
|            |           |      x        |            |
> | `DEBIAN_DEPENDS`   |            |           |               |     x      |

While this graphic looks very simple, that is misguiding.
IMO they could lead to a viewpoint that makes really understanding what 
is happening more difficult. Bitbake works its dependency magic on task 
level and provides DEPENDS to express common task dependencies across 
recipes as syntax sugar.

IMAGE_INSTALL + IMAGE_PREINSTALL + DEBIAN_DEPENDS etc. is more about 
expressing debian package installation and a bit more. This time it is 
isar syntax sugar that puts the entries from IMAGE_INSTALL into DEPENDS.

IMO this sugar, while it makes some things more easy (with big 
assumptions), makes some things more difficult (recipes provide multiple 
packages). AFAIK this wasn't changed because historic reasons.

> 
> ## How to control buildtime dependencies
> 
> As mentioned above, buildtime dependencies specify which Bitbake recipes get executed and Debian packages get built.
> 
> The union of recipes specified via `DEPENDS` and `IMAGE_INSTALL` and after applying filters like `PREFERRED_VERSION` gets selected to be [executed][bb-tasks].
> 
> ### Bitbake buildtime: `DEPENDS`
> 
> `DEPENDS` specifies recipes available in one of the layers that have to be built.
> 
> The logical place to use this variable depends on which kind of dependency it should specify.
> 
> Is it a recipe that builds a package that requires another custom package (built from a Bitbake recipe)?
> Then probably a **package** recipe is the best place. > The [above table](#dependencies-table) shows that there is no way to 
specify at the same time a Bitbake runtime dependency and a Debian 
Package dependency, therefore the corresponding [package runtime
> dependency](#package-runtime-dependencies) should be also added.
> 
> Otherwise probably an **image** recipe or **configuration** file is the best place.

No. DEPENDS in a configuration file does not make any sense to me. 
Configuration files are global, that would mean every recipes would have 
this dependency now (if it doesn't just overwrite it)

And for Isar DEPENDS in a image recipe is very obscure as well, see 
IMAGER_BUILD_DEPS, where it is used.

DEPENDS also has nothing to do with packages directly but with the vague 
dependencies between recipes that then get concrete transferred into 
dependencies of tasks between recipes.

See:

https://www.yoctoproject.org/docs/latest/bitbake-user-manual/bitbake-user-manual.html#build-dependencies

> Opposed to `IMAGE_INSTALL`, this doesn't affect any runtime dependencies.
> 
> This variable can be seen as the counterpart at buildtime for [`IMAGE_PREINSTALL`](#functional-runtime-image_preinstall) (runtime) and [`DEBIAN_DEPENDS`](#package-runtime-debian_depends) (runtime).
> 
> ### Bitbake buildtime: `IMAGE_INSTALL`
> 
> `IMAGE_INSTALL` specifies recipes available in one of the layers that have to be built. > And at the same time specifies that the Debian packages resulting 
from the recipes have to be installed in the image (see 
[below](#functional-runtime-image_install) for more information).
> 
> The logical place to use this variable is an **image** recipe or **configuration** file (distro, layer, ...).

The only places where this is possible.

Speaking of images, there are also 'debian build dependencies' for 
images called 'IMAGER_INSTALL'.

> 
> ## How to control runtime dependencies
> 
> Runtime dependencies specify which Debian packages are made availabe during runtime (by installing them in the image, if one is being created).
> 
> The union of packages specified via `IMAGE_INSTALL`, `IMAGE_PREINSTALL`, `DEBIAN_DEPENDS`, `debian/control` files of custom packages and dependencies of upstream Debian packages get installed into the
> image.

+ the minbase that debootstrap installs.

> 
> ### Functional runtime: `IMAGE_INSTALL`
> 
> `IMAGE_INSTALL` specifies custom Debian packages that have to be installed in the image.
> And at the same time specifies that the correponding Bitbake recipes have to be previously executed to build the Debian packages (see [above](#bitbake-buildtime-image_install) for more information).
> 
> The logical place to use this variable is an **image** recipe or **configuration** file (distro, layer, ...).

See comment above

> 
> ### Functional runtime: `IMAGE_PREINSTALL`
> 
> `IMAGE_PREINSTALL` specifies upstream Debian packages (opposed to custom Debian packages built from Bitbake recipes) that are needed in the image.
> 
> The logical place to use this variable is an **image** recipe or **configuration** file (distro, layer, ...).

Same as IMAGE_INSTALL comment.

> 
> These packages will be marked as "manually installed" (they can be shown with `apt-mark showmanual`).
> 
> Opposed to `IMAGE_INSTALL`, this doesn't affect any dependencies at buildtime.
> 
> This variable can be seen as one counterpart at runtime for [`DEPENDS`](#bitbake-buildtime-depends) at buildtime.
> 
> ### Package runtime: `DEBIAN_DEPENDS`
> 
> `DEBIAN_DEPENDS` specifies the packages that are required by the package being built.
> They will be added to the [`Depends` of the package control file][pkg-rels] and therefore have to be specified in the format expected by Debian.
> 
> The logical place to use this variable is a **package** recipe.

Only place are dpkg-raw packages that are package recipes.

> 
> These packages will be marked as "automatically installed" (they can be shown with `apt-mark showauto`).
> 
> Opposed to `IMAGE_INSTALL`, this doesn't affect any dependencies at builtime.
> 
> This variable can be seen as one counterpart at runtime for [`DEPENDS`](#bitbake-buildtime-depends) at buildtime.
> 
> ## Bitbake variables being ignored by ISAR
> 
> This section is documenting Bitbake variables being used for package definition and dependency specification in OpenEmbedded and Yocto, but being ignored by ISAR!
> 
> The ISAR ways to accomplish the same goal are being also documented here.
> 
> ### `RDEPENDS`
> 
> This variable ["lists a package's runtime dependencies"][bb-rdepends].
> 
> In ISAR there are two ways to achieve the same:
> 
> 1. Specify the dependencies via [`DEBIAN_DEPENDS`](#package-runtime-debian_depends) if inheriting from the [`dpkg-raw` class][dpkg-raw]].
> 1. Adding them to the [`Depends` of the package `debian/control`][pkg-rels] file.
> 
> ### `RRECOMMENDS`
> 
> This variable provides ["a list of packages that extends the usability of a package"][bb-rrecommends].
> 
> In ISAR there is only one way to achieve the same:
> 
> 1. Adding them to the [`Recommends` of the package `debian/control`][pkg-rels] file.

You can do this, but recommended or suggested packages are not installed 
per default, but you could change that by overwriting the isar-apt.conf 
of the isar-bootstrap-*.bb recipes.

> 
> ### `PACKAGES`
> 
> This variable provides ["the list of packages the recipe creates"][bb-packages].
> 
> In ISAR there is only one way to achieve the same:
> 
> 1. Specifying multiple packages in the [`debian/control` file][pkg-control] being provided by a recipe.

Doing so might get problematic because you then cannot use IMAGE_INSTALL 
to install those specific packages, because they then will be added to 
DEPENDS (as I said IMO bad idea) and have to remove those from DEPENDS 
(DEPENDS_remove) and add the correct recipe name manually.

regards,
Claudius

> 
> [pkg-rels]: https://www.debian.org/doc/debian-policy/ch-relationships.html "Debian policy: Declaring relationships between packages"
> [pkg-control]: https://www.debian.org/doc/debian-policy/ch-controlfields.html "Debian policy: Control files and their fields"
> [dpkg-raw]: https://github.com/ilbers/isar/blob/6c5db020b9b837d7b0ce63bfc719f9192e725f26/meta/classes/dpkg-raw.bbclass
> [bb-tasks]: https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html#executing-tasks "Bitbake: Executing Tasks"
> [bb-rdepends]: https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html#var-RDEPENDS "Bitbake: RDEPENDS"
> [bb-rrecommends]: https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html#var-RRECOMMENDS "Bitbake: RRECOMMENDS"
> [bb-packages]: https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html#var-PACKAGES "Bitbake: PACKAGES"
> 

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [RFC] Documentation: dependencies specification in Bitbake and Debian

[Cirujano Cuesta, Silvano]

Hi Claudius,

First of all thank you very much for your quick and valuable feedback!

On Wed, 2019-02-06 at 16:00 +0100, Claudius Heine wrote:
> > ## Dependencies table
> > 
> > Acronyms:
> > > Bt = Buildtime
> > > 
> > > Rt = Runtime
> > > Variable           | Bt Bitbake | Bt Debian | Rt Functional | Rt Package |
> > > ------------------ |:----------:|:---------:|:-------------:|:----------:|
> > > `DEPENDS`          |     x      |           |               |            |
> > > `IMAGE_INSTALL`    |     x      |           |      x        |            | > | `IMAGE_PREINSTALL` 
> > 
> >             |           |      x        |            |
> > > `DEBIAN_DEPENDS`   |            |           |               |     x      |
> 
> While this graphic looks very simple, that is misguiding.
> IMO they could lead to a viewpoint that makes really understanding what 
> is happening more difficult. Bitbake works its dependency magic on task 
> level and provides DEPENDS to express common task dependencies across 
> recipes as syntax sugar.
> 
> IMAGE_INSTALL + IMAGE_PREINSTALL + DEBIAN_DEPENDS etc. is more about 
> expressing debian package installation and a bit more. This time it is 
> isar syntax sugar that puts the entries from IMAGE_INSTALL into DEPENDS.
> 
> IMO this sugar, while it makes some things more easy (with big 
> assumptions), makes some things more difficult (recipes provide multiple 
> packages). AFAIK this wasn't changed because historic reasons.

I'm not sure I get your point here...

My interpretation of your comment is that the table is right, but you would prefer the table not to show the syntactic sugar.
Mostly to avoid people getting into anti-patterns, I assume.
Did I get it right?

The separation between Bitbake dependency at task level and Debian dependency at package level is clear to me, perhaps it's only the presentation of the table that is misleading...

> > ### Bitbake buildtime: `DEPENDS`
> > 
> > `DEPENDS` specifies recipes available in one of the layers that have to be built.
> > 
> > The logical place to use this variable depends on which kind of dependency it should specify.
> > 
> > Is it a recipe that builds a package that requires another custom package (built from a Bitbake recipe)?
> > Then probably a **package** recipe is the best place. > The [above table](#dependencies-table) shows that there is no way to 
> 
> specify at the same time a Bitbake runtime dependency and a Debian 
> Package dependency, therefore the corresponding [package runtime
> > dependency](#package-runtime-dependencies) should be also added.
> > 
> > Otherwise probably an **image** recipe or **configuration** file is the best place.
> 
> No. DEPENDS in a configuration file does not make any sense to me. 
> Configuration files are global, that would mean every recipes would have 
> this dependency now (if it doesn't just overwrite it)

You are right. I will fix it.

> 
> And for Isar DEPENDS in a image recipe is very obscure as well, see 
> IMAGER_BUILD_DEPS, where it is used.

WHAT?! I haven't found any reference to `IMAGER_BUILD_DEPS` anywhere in the documentation!

Looking at the code [1] I see that it's an obvious miss in the documentation :-)
Damn it, I'll have to analyze that code if I want this documentation to be complete...

[1] https://github.com/ilbers/isar/blob/6c5db020b9b837d7b0ce63bfc719f9192e725f26/meta/classes/image.bbclass#L131

You are anyway right, that an image recipe is an obscure place for a DEPENDS.
The right place would be a package recipe. I will fix it.

> 
> DEPENDS also has nothing to do with packages directly but with the vague 
> dependencies between recipes that then get concrete transferred into 
> dependencies of tasks between recipes.
> 
> See:
> 
> https://www.yoctoproject.org/docs/latest/bitbake-user-manual/bitbake-user-manual.html#build-dependencies

That's clear to me, apparently the formulation in the text doesn't express it clearly enough.

> 
> > Opposed to `IMAGE_INSTALL`, this doesn't affect any runtime dependencies.
> > 
> > This variable can be seen as the counterpart at buildtime for [`IMAGE_PREINSTALL`](#functional-runtime-image_preinstall) (runtime) and [`DEBIAN_DEPENDS`](#package-runtime-debian_depends)
> > (runtime).
> > 
> > ### Bitbake buildtime: `IMAGE_INSTALL`
> > 
> > `IMAGE_INSTALL` specifies recipes available in one of the layers that have to be built. > And at the same time specifies that the Debian packages resulting 
> 
> from the recipes have to be installed in the image (see 
> [below](#functional-runtime-image_install) for more information).
> > 
> > The logical place to use this variable is an **image** recipe or **configuration** file (distro, layer, ...).
> 
> The only places where this is possible.

Does "is possible" really mean that either you get an error or it's being silently ignored?
Or there is no enforcement, but it shouldn't be done elsewhere?

> 
> Speaking of images, there are also 'debian build dependencies' for 
> images called 'IMAGER_INSTALL'.

Another surprise! I haven't found any reference either to it anywhere in the documentation!
The same as `IMAGER_BUILD_DEPS`.

> 
> > 
> > ## How to control runtime dependencies
> > 
> > Runtime dependencies specify which Debian packages are made availabe during runtime (by installing them in the image, if one is being created).
> > 
> > The union of packages specified via `IMAGE_INSTALL`, `IMAGE_PREINSTALL`, `DEBIAN_DEPENDS`, `debian/control` files of custom packages and dependencies of upstream Debian packages get installed into
> > the
> > image.
> 
> + the minbase that debootstrap installs.

True. I will add it.

> > ### Package runtime: `DEBIAN_DEPENDS`
> > 
> > `DEBIAN_DEPENDS` specifies the packages that are required by the package being built.
> > They will be added to the [`Depends` of the package control file][pkg-rels] and therefore have to be specified in the format expected by Debian.
> > 
> > The logical place to use this variable is a **package** recipe.
> 
> Only place are dpkg-raw packages that are package recipes.

Right. Too much copy&paste :-)

> > ### `RRECOMMENDS`
> > 
> > This variable provides ["a list of packages that extends the usability of a package"][bb-rrecommends].
> > 
> > In ISAR there is only one way to achieve the same:
> > 
> > 1. Adding them to the [`Recommends` of the package `debian/control`][pkg-rels] file.
> 
> You can do this, but recommended or suggested packages are not installed 
> per default, but you could change that by overwriting the isar-apt.conf 
> of the isar-bootstrap-*.bb recipes.

That's clear to me in the case of Debian.
I've also assumed that the same applies to Yocto/OE

> 
> > 
> > ### `PACKAGES`
> > 
> > This variable provides ["the list of packages the recipe creates"][bb-packages].
> > 
> > In ISAR there is only one way to achieve the same:
> > 
> > 1. Specifying multiple packages in the [`debian/control` file][pkg-control] being provided by a recipe.
> 
> Doing so might get problematic because you then cannot use IMAGE_INSTALL 
> to install those specific packages, because they then will be added to 
> DEPENDS (as I said IMO bad idea) and have to remove those from DEPENDS 
> (DEPENDS_remove) and add the correct recipe name manually.

Well, `DEPENDS` would be needed to get the "Buildtime dependency" (to ensure that the recipe gets executed and the Debian packages get built) and the control file would take care of the "Runtime
dependency" (to get the functionality in the image or a dependency of a package resolved).

I'm not sure what you mean it's a bad idea.
Hopefully you don't mean that having `DEPENDS` in a package recipe is a bad idea, since that's what the example does [2].

[2] https://github.com/ilbers/isar/blob/6c5db020b9b837d7b0ce63bfc719f9192e725f26/meta-isar/recipes-app/example-hello/example-hello.bb#L15

Regards,
  Silvano

Re: [RFC] Documentation: dependencies specification in Bitbake and Debian

[Claudius Heine]

[-- Attachment #1: Type: text/plain, Size: 7320 bytes --]

Hi Silvano,

Quoting Cirujano Cuesta, Silvano (2019-02-06 17:11:40)
> Hi Claudius,
> 
> First of all thank you very much for your quick and valuable feedback!
> 
> On Wed, 2019-02-06 at 16:00 +0100, Claudius Heine wrote:
> > > ## Dependencies table
> > > 
> > > Acronyms:
> > > > Bt = Buildtime
> > > > 
> > > > Rt = Runtime
> > > > Variable           | Bt Bitbake | Bt Debian | Rt Functional | Rt Package |
> > > > ------------------ |:----------:|:---------:|:-------------:|:----------:|
> > > > `DEPENDS`          |     x      |           |               |            |
> > > > `IMAGE_INSTALL`    |     x      |           |      x        |            | > | `IMAGE_PREINSTALL` 
> > > 
> > >             |           |      x        |            |
> > > > `DEBIAN_DEPENDS`   |            |           |               |     x      |
> > 
> > While this graphic looks very simple, that is misguiding.
> > IMO they could lead to a viewpoint that makes really understanding what 
> > is happening more difficult. Bitbake works its dependency magic on task 
> > level and provides DEPENDS to express common task dependencies across 
> > recipes as syntax sugar.
> > 
> > IMAGE_INSTALL + IMAGE_PREINSTALL + DEBIAN_DEPENDS etc. is more about 
> > expressing debian package installation and a bit more. This time it is 
> > isar syntax sugar that puts the entries from IMAGE_INSTALL into DEPENDS.
> > 
> > IMO this sugar, while it makes some things more easy (with big 
> > assumptions), makes some things more difficult (recipes provide multiple 
> > packages). AFAIK this wasn't changed because historic reasons.
> 
> I'm not sure I get your point here...
> 
> My interpretation of your comment is that the table is right, but you would prefer the table not to show the syntactic sugar.
> Mostly to avoid people getting into anti-patterns, I assume.
> Did I get it right?
> 
> The separation between Bitbake dependency at task level and Debian dependency at package level is clear to me, perhaps it's only the presentation of the table that is misleading...

In general you would need 3 such tables. Two for package recipes (one
for dpkg and one for dpkg-raw) and one for image recipes.

My main issue is that you are comparing things here from different
classes and bitbake in general. While the table is correct it is
misleading IMO.

[...]

> > 
> > > Opposed to `IMAGE_INSTALL`, this doesn't affect any runtime dependencies.
> > > 
> > > This variable can be seen as the counterpart at buildtime for [`IMAGE_PREINSTALL`](#functional-runtime-image_preinstall) (runtime) and [`DEBIAN_DEPENDS`](#package-runtime-debian_depends)
> > > (runtime).
> > > 
> > > ### Bitbake buildtime: `IMAGE_INSTALL`
> > > 
> > > `IMAGE_INSTALL` specifies recipes available in one of the layers that have to be built. > And at the same time specifies that the Debian packages resulting 
> > 
> > from the recipes have to be installed in the image (see 
> > [below](#functional-runtime-image_install) for more information).
> > > 
> > > The logical place to use this variable is an **image** recipe or **configuration** file (distro, layer, ...).
> > 
> > The only places where this is possible.
> 
> Does "is possible" really mean that either you get an error or it's being silently ignored?
> Or there is no enforcement, but it shouldn't be done elsewhere?

It is silently ignored if it not part of an image recipe or the global
configuration. Each recipe runs with its own copy of the bitbake data
store, so you cannot effect what goes into an image via IMAGE_INSTALL
from a package recipe for instance.

[...]

> > > ### `RRECOMMENDS`
> > > 
> > > This variable provides ["a list of packages that extends the usability of a package"][bb-rrecommends].
> > > 
> > > In ISAR there is only one way to achieve the same:
> > > 
> > > 1. Adding them to the [`Recommends` of the package `debian/control`][pkg-rels] file.
> > 
> > You can do this, but recommended or suggested packages are not installed 
> > per default, but you could change that by overwriting the isar-apt.conf 
> > of the isar-bootstrap-*.bb recipes.
> 
> That's clear to me in the case of Debian.
> I've also assumed that the same applies to Yocto/OE

Well I haven't tried that, but I suppose that you can enable recommended
packages just via a configuration setting. Not so here.

> > > ### `PACKAGES`
> > > 
> > > This variable provides ["the list of packages the recipe creates"][bb-packages].
> > > 
> > > In ISAR there is only one way to achieve the same:
> > > 
> > > 1. Specifying multiple packages in the [`debian/control` file][pkg-control] being provided by a recipe.
> > 
> > Doing so might get problematic because you then cannot use IMAGE_INSTALL 
> > to install those specific packages, because they then will be added to 
> > DEPENDS (as I said IMO bad idea) and have to remove those from DEPENDS 
> > (DEPENDS_remove) and add the correct recipe name manually.
> 
> Well, `DEPENDS` would be needed to get the "Buildtime dependency" (to ensure that the recipe gets executed and the Debian packages get built) and the control file would take care of the "Runtime
> dependency" (to get the functionality in the image or a dependency of a package resolved).
> 
> I'm not sure what you mean it's a bad idea.
> Hopefully you don't mean that having `DEPENDS` in a package recipe is a bad idea, since that's what the example does [2].

No. I called inserting IMAGE_INSTALL into the DEPENDS of an image was a
bad idea when that was done, because that assumes that a recipe provides
just one package with the same name. If you want for instance a "curl"
recipe to generate a "libcurl" package and you want to install just
that, you would have to add 'curl' into DEPENDS and 'libcurl' into
IMAGE_INSTALL. But because IMAGE_INSTALL now adds 'libcurl' into DEPENDS
and there is no 'libcurl' recipe, you would have to remove 'libcurl'
from DEPENDS manually.

regards,
Claudius

> 
> [2] https://github.com/ilbers/isar/blob/6c5db020b9b837d7b0ce63bfc719f9192e725f26/meta-isar/recipes-app/example-hello/example-hello.bb#L15
> 
> Regards,
>   Silvano
> 
> -- 
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
> To post to this group, send email to isar-users@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/1549469493.6521.19.camel%40siemens.com.
> For more options, visit https://groups.google.com/d/optout.

--
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

           PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                             Keyserver: hkp://pool.sks-keyservers.net

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEb/LlnwDGvCgx2GTBEXPLGZgIsVMFAlxbIq8ACgkQEXPLGZgI
sVMEWw//SVUtnAF9gud5ICaIIZJ2I2fK5OIzDPgFnIm4p6XAHVVqsUPGK0135RGC
0mMii/UtNef9uFyrc97eGWFI3u9KVSXaQn4kCnRbPiUpqOGnsc56Wrb+syeh/n3l
GXlfKYpSY5+fgNSwhhLvl4WHok6OOLgzm5crbgTNxWBFUkOVkUmyHD3DGflD5OyG
M9O8QLeru0xt770QjlkBLk9f6htqJ7rvClDvkgCRoXHqhgWpQTTgbs5pdU8/3hWW
uNhO4LfcOUQb0gKgaJ9TeZCL1U2anJTg9DHQBzwWPZsMES5LY3NHWNt4J/Z/SLfN
Mn+O8BOOhQB1Lu5JKeIrBJ44IS9jk2wO8VOfGFoHLwzssaTE1NS1xbo6nmT7sj08
H0J8LKd1ADnnXvXwOyZoozZ3qALRRN47CZd5LCCfVNsjzNT4V2VPZp1JFFOUqpRS
vk0S9Mgz+rd7VNzjMVfiMhl+VpwlZmAVhwmoOHGbtuUM0XBInjT1hsl/U1GevNtQ
l8W8PD7zhdVqQF5ioQE2SChoYoFtbrT+Pr2r1hv7lS1cWng8P+qx/FXaYEcmphhE
QQetx719pBYjBG5DK32Irb7wE0U1YDb148hiar8CAaAdKT/vVru0AFHG0Acq3Ot0
fIrbP+YoUQbMwf4GaamdmIP+MO3RojcMit6ZQGjjF8C0v1UGqk8=
=1k2X
-----END PGP SIGNATURE-----

Re: [RFC] Documentation: dependencies specification in Bitbake and Debian

[Claudius Heine]

[-- Attachment #1: Type: text/plain, Size: 1490 bytes --]

Hi Silvano,

Quoting Claudius Heine (2019-02-06 19:08:51)
> > > > ### `RRECOMMENDS`
> > > > 
> > > > This variable provides ["a list of packages that extends the usability of a package"][bb-rrecommends].
> > > > 
> > > > In ISAR there is only one way to achieve the same:
> > > > 
> > > > 1. Adding them to the [`Recommends` of the package `debian/control`][pkg-rels] file.
> > > 
> > > You can do this, but recommended or suggested packages are not installed 
> > > per default, but you could change that by overwriting the isar-apt.conf 
> > > of the isar-bootstrap-*.bb recipes.
> > 
> > That's clear to me in the case of Debian.
> > I've also assumed that the same applies to Yocto/OE
> 
> Well I haven't tried that, but I suppose that you can enable recommended
> packages just via a configuration setting. Not so here.

I looked into that. In OE recommended packages are installed per
default, but that can be globally (NO_RECOMMENDATIONS) or selectivly
(BAD_RECOMMENDATIONS) changed. In Isar they are disabled by default and
there is currently no simple switch for that.

regards,
Claudius

--
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

           PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                             Keyserver: hkp://pool.sks-keyservers.net

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEb/LlnwDGvCgx2GTBEXPLGZgIsVMFAlxdShkACgkQEXPLGZgI
sVMJqxAArcdVFVLo/ijKP03P0WUU+VyGh32LE018LNEs4C17PceT//ISTl5dnPh+
hRIhgZt2WxYnFBZTx48Rdh+/pga/qwuxCCeCMUhH5zyIEZm8E1HIRf9xpX1Hs0Pc
YoyIQ6N+lEWJIjBs6npR59OXdFl62KOJWx9pfTFu+cztWqHorBin/QjWv0onYYDA
mSvSZwWRT9WJ++cprGoRi0vGeU0o++HXzuziN7C6zPQ13eBZADaZczVFXuV4uV5l
fHmdvWaaGWbhsdyMLVfCWYimCI3M6dpbmydvNKm9wfVv+sY9bTbKR2sbdPASC95R
B7RQAXZzkb7Vlqo04x6SGEleg418/hL0gfpCvehcvIbIu5cUrerdg57C2M66OtMc
O/IKl+AO/snbbbpsTdwYRlUQpUv7zIuObN3rsHbO6TRJoIaPc7GzpCZ8I8LahUwM
BcPIJBU8LeXHeSrtpUGmoWNF4ZECZfH1PUuVLFos8eEehLb+/1hNhUshnp9di/+p
ZSTK1PbBC8DTJnSxSg3YcEjJkzDRXSuDSwcMN9/csTTo5zsIsR3JUtoB1oS/LdIp
DvoHwRcW3qplRc9Nxrg/7J9XjZ5djRZG6cg455KxpwJ8ZPz1diBYqi3Zqc/DJvoY
Mifh9ZoNrQHSo69K4s2Ez+LsTaoxkKVouhECpf+sOE9cTcG5v7U=
=+MtD
-----END PGP SIGNATURE-----