From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6654880414262362112 X-Received: by 2002:a2e:4253:: with SMTP id p80-v6mr744693lja.13.1549465209010; Wed, 06 Feb 2019 07:00:09 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:ac2:4296:: with SMTP id m22ls594894lfh.3.gmail; Wed, 06 Feb 2019 07:00:08 -0800 (PST) X-Google-Smtp-Source: AHgI3IYpxF7kwVok+j3n3s5DurAWTF4ujCHAadUtc9YWNVYPnZ6pS8aJ+65WhdsWmgAZHV6773qO X-Received: by 2002:a19:5519:: with SMTP id n25mr728631lfe.1.1549465208394; Wed, 06 Feb 2019 07:00:08 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1549465208; cv=none; d=google.com; s=arc-20160816; b=zUKwC82TUGBSHP3FAKeyQBK2cZAglUxKJVmFMHTPsJCbLptDz3N31NmwxbyoAlPHPI e3xNXDP29ggov5YVf8LzmEOsv6OFMZHHE7cg1VjvJ8lM5WzNiCK+4EPmXl0nL8FWW188 /QB++BjkPn/oY+jSZn292Mid+2VDBK35iZb6aeJoprgxFx0bXyd5JxsnJNWPUrGqqXGe q35dk0V38OsbX1UP2gmd0adgtvldmTmEBwdJu6Zx+INpHKfXa87f75TUYeickpuS87It LzvVFl+NOxOetuB7XgUeiJeofcbQI4M+o9ADBQf735RW/3JThdKI/+LchWl5sy3PqQEp wOYg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject; bh=uz+BfOgqf7xcAN/VWvrfVTB4vp2PaRuz3rgoKmbODgA=; b=tMlmq1SCvD/mwY/8WO0LtldzVB19kNg9cVpA/EYWJh+bnnkEtnhsdUpR3e8tcfl1CA QVA06mppFuXC6n5H3/MTyo13jd815joJTr9YKePJ15JRp2STyjMlWNsOw/oC694OLkpl jLCsgLSyMdrVNvZEjd4IaQ5Qk+nW+AGJxUuT+7F1pAVOfya0YhKuOQbEO//9V838O8ND rnCK7FTqI5c0w6079TBT0REjS6/1j5Ft0CWjU9GaGbvRGTysMQBuDmrn4UJ3cvgpZjp1 sWnjVEHzsoTM248SPrEQ0xfqaaJaeATQKBlJqR7UEUsawanfcJCzUBI9sPiz8KX6T+4i ec8Q== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com Return-Path: Received: from thoth.sbs.de (thoth.sbs.de. [192.35.17.2]) by gmr-mx.google.com with ESMTPS id y18si1103536lfe.3.2019.02.06.07.00.08 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 06 Feb 2019 07:00:08 -0800 (PST) Received-SPF: pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.2 as permitted sender) client-ip=192.35.17.2; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by thoth.sbs.de (8.15.2/8.15.2) with ESMTPS id x16F07V5016037 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Wed, 6 Feb 2019 16:00:07 +0100 Received: from [139.25.69.181] (linux-ses-ext02.ppmd.siemens.net [139.25.69.181]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id x16F07XV018868; Wed, 6 Feb 2019 16:00:07 +0100 Subject: Re: [RFC] Documentation: dependencies specification in Bitbake and Debian To: "[ext] Cirujano Cuesta, Silvano" , "isar-users@googlegroups.com" References: <1549459957.6521.14.camel@siemens.com> From: Claudius Heine Message-ID: <7ed827ce-2243-4341-ee61-6d3e3881bb4a@siemens.com> Date: Wed, 6 Feb 2019 16:00:07 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 In-Reply-To: <1549459957.6521.14.camel@siemens.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: F9tfEKqw50Xk Hi Silvano On 06/02/2019 14.34, [ext] Cirujano Cuesta, Silvano wrote: > Hi, > > I've tried to document my understanding on dependency management at different areas in ISAR. > I've done it mostly to structure it in my head. > But I'm also convinced that this kind of documentation is critical for lowering the entry barrier to ISAR usage, therefore I'm sending it to this mailing list. > > With this request for comments I want to: > 1. gather comments about the demand for such documentation Yes that would be good to have. IMO > 2. gather comments about the content to make sure that I didn't misunderstood how it works See below. > > Once I get confirmation for the demand and any possible misconception clarified, I'll provide a patch integrating it with the already existing documentation. > > Please keep comments about the container in the chamber. You can shoot them later when I send the patch ;-) > > You can find the content in Markdown format at the bottom. > > Regards, > Silvano > > -------------------------------- > > # Dependency management in ISAR > > ## Dependencies resolution > > Dependencies have to be taken into account separately on different stages: > > 1. Builtime dependencies >    1. [Bitbake buildtime dependencies](#bitbake-buildtime-dependencies) >    1. [Debian buildtime dependencies](#debian-buildtime-dependencies) > 1. Runtime dependencies >    1. [Functional runtime dependencies](#functional-runtime-dependencies) >    1. [Package runtime dependencies](#package-runtime-dependencies) > > ### Bitbake buildtime dependencies > > The execution of a Bitbake recipe might require the execution of other recipes, these are the Bitbake buildtime dependencies. > > ### Debian buildtime dependencies > > Debian is not only the distribution of the resulting root filesystem, but also the distribution of the build environment. > Debian buildtime dependencies are Debian packages needed in the build environment to build certain Debian package. > > Following sections show that ISAR doesn't provide any automatic mechanism to specify these dependencies. > Therefore [`Build-Depends`][pkg-rels] has to be specified in the corresponding Debian Control file (`debian/control`). > > ### Functional runtime dependencies > > These are Debian packages needed to provide certain functionality in the target. > > They are explicitly installed in the image, therefore will be considered as "manually" installed by APT, as a consequence they can only be removed manually. > > Functional runtime dependencies can be seen as image dependencies, since they are required for the image to provide the expected functionality. > > ### Package runtime dependencies > > These are Debian packages needed by packages being "manually" installed. They are not needed by themselves, but only to obtain the functionality provided by the [functional runtime > dependencies](#functional-runtime-dependencies) > > Upstream Debian packages already provide their own package runtime dependencies. > > Custom packages have to provide these dependencies. They can be specified via the corresponding Debian Control files (`debian/control`). > > ISAR supports on specifying these dependencies if inheriting from the [`dpkg-raw` class][dpkg-raw]. > > These packages will be considered as "automatically" installed by APT as dependencies of other packages, as a consequence APT can identify when they are not needed anymore. > APT provides the subcommand `autoremove` to uninstall them. > > ## Dependencies table > > Acronyms: >> Bt = Buildtime >> >> Rt = Runtime > > | Variable           | Bt Bitbake | Bt Debian | Rt Functional | Rt Package | > | ------------------ |:----------:|:---------:|:-------------:|:----------:| > | `DEPENDS`          |     x      |           |               |            | > | `IMAGE_INSTALL`    |     x      |           |      x        |            | > | `IMAGE_PREINSTALL` |            |           |      x        |            | > | `DEBIAN_DEPENDS`   |            |           |               |     x      | While this graphic looks very simple, that is misguiding. IMO they could lead to a viewpoint that makes really understanding what is happening more difficult. Bitbake works its dependency magic on task level and provides DEPENDS to express common task dependencies across recipes as syntax sugar. IMAGE_INSTALL + IMAGE_PREINSTALL + DEBIAN_DEPENDS etc. is more about expressing debian package installation and a bit more. This time it is isar syntax sugar that puts the entries from IMAGE_INSTALL into DEPENDS. IMO this sugar, while it makes some things more easy (with big assumptions), makes some things more difficult (recipes provide multiple packages). AFAIK this wasn't changed because historic reasons. > > ## How to control buildtime dependencies > > As mentioned above, buildtime dependencies specify which Bitbake recipes get executed and Debian packages get built. > > The union of recipes specified via `DEPENDS` and `IMAGE_INSTALL` and after applying filters like `PREFERRED_VERSION` gets selected to be [executed][bb-tasks]. > > ### Bitbake buildtime: `DEPENDS` > > `DEPENDS` specifies recipes available in one of the layers that have to be built. > > The logical place to use this variable depends on which kind of dependency it should specify. > > Is it a recipe that builds a package that requires another custom package (built from a Bitbake recipe)? > Then probably a **package** recipe is the best place. > The [above table](#dependencies-table) shows that there is no way to specify at the same time a Bitbake runtime dependency and a Debian Package dependency, therefore the corresponding [package runtime > dependency](#package-runtime-dependencies) should be also added. > > Otherwise probably an **image** recipe or **configuration** file is the best place. No. DEPENDS in a configuration file does not make any sense to me. Configuration files are global, that would mean every recipes would have this dependency now (if it doesn't just overwrite it) And for Isar DEPENDS in a image recipe is very obscure as well, see IMAGER_BUILD_DEPS, where it is used. DEPENDS also has nothing to do with packages directly but with the vague dependencies between recipes that then get concrete transferred into dependencies of tasks between recipes. See: https://www.yoctoproject.org/docs/latest/bitbake-user-manual/bitbake-user-manual.html#build-dependencies > Opposed to `IMAGE_INSTALL`, this doesn't affect any runtime dependencies. > > This variable can be seen as the counterpart at buildtime for [`IMAGE_PREINSTALL`](#functional-runtime-image_preinstall) (runtime) and [`DEBIAN_DEPENDS`](#package-runtime-debian_depends) (runtime). > > ### Bitbake buildtime: `IMAGE_INSTALL` > > `IMAGE_INSTALL` specifies recipes available in one of the layers that have to be built. > And at the same time specifies that the Debian packages resulting from the recipes have to be installed in the image (see [below](#functional-runtime-image_install) for more information). > > The logical place to use this variable is an **image** recipe or **configuration** file (distro, layer, ...). The only places where this is possible. Speaking of images, there are also 'debian build dependencies' for images called 'IMAGER_INSTALL'. > > ## How to control runtime dependencies > > Runtime dependencies specify which Debian packages are made availabe during runtime (by installing them in the image, if one is being created). > > The union of packages specified via `IMAGE_INSTALL`, `IMAGE_PREINSTALL`, `DEBIAN_DEPENDS`, `debian/control` files of custom packages and dependencies of upstream Debian packages get installed into the > image. + the minbase that debootstrap installs. > > ### Functional runtime: `IMAGE_INSTALL` > > `IMAGE_INSTALL` specifies custom Debian packages that have to be installed in the image. > And at the same time specifies that the correponding Bitbake recipes have to be previously executed to build the Debian packages (see [above](#bitbake-buildtime-image_install) for more information). > > The logical place to use this variable is an **image** recipe or **configuration** file (distro, layer, ...). See comment above > > ### Functional runtime: `IMAGE_PREINSTALL` > > `IMAGE_PREINSTALL` specifies upstream Debian packages (opposed to custom Debian packages built from Bitbake recipes) that are needed in the image. > > The logical place to use this variable is an **image** recipe or **configuration** file (distro, layer, ...). Same as IMAGE_INSTALL comment. > > These packages will be marked as "manually installed" (they can be shown with `apt-mark showmanual`). > > Opposed to `IMAGE_INSTALL`, this doesn't affect any dependencies at buildtime. > > This variable can be seen as one counterpart at runtime for [`DEPENDS`](#bitbake-buildtime-depends) at buildtime. > > ### Package runtime: `DEBIAN_DEPENDS` > > `DEBIAN_DEPENDS` specifies the packages that are required by the package being built. > They will be added to the [`Depends` of the package control file][pkg-rels] and therefore have to be specified in the format expected by Debian. > > The logical place to use this variable is a **package** recipe. Only place are dpkg-raw packages that are package recipes. > > These packages will be marked as "automatically installed" (they can be shown with `apt-mark showauto`). > > Opposed to `IMAGE_INSTALL`, this doesn't affect any dependencies at builtime. > > This variable can be seen as one counterpart at runtime for [`DEPENDS`](#bitbake-buildtime-depends) at buildtime. > > ## Bitbake variables being ignored by ISAR > > This section is documenting Bitbake variables being used for package definition and dependency specification in OpenEmbedded and Yocto, but being ignored by ISAR! > > The ISAR ways to accomplish the same goal are being also documented here. > > ### `RDEPENDS` > > This variable ["lists a package's runtime dependencies"][bb-rdepends]. > > In ISAR there are two ways to achieve the same: > > 1. Specify the dependencies via [`DEBIAN_DEPENDS`](#package-runtime-debian_depends) if inheriting from the [`dpkg-raw` class][dpkg-raw]]. > 1. Adding them to the [`Depends` of the package `debian/control`][pkg-rels] file. > > ### `RRECOMMENDS` > > This variable provides ["a list of packages that extends the usability of a package"][bb-rrecommends]. > > In ISAR there is only one way to achieve the same: > > 1. Adding them to the [`Recommends` of the package `debian/control`][pkg-rels] file. You can do this, but recommended or suggested packages are not installed per default, but you could change that by overwriting the isar-apt.conf of the isar-bootstrap-*.bb recipes. > > ### `PACKAGES` > > This variable provides ["the list of packages the recipe creates"][bb-packages]. > > In ISAR there is only one way to achieve the same: > > 1. Specifying multiple packages in the [`debian/control` file][pkg-control] being provided by a recipe. Doing so might get problematic because you then cannot use IMAGE_INSTALL to install those specific packages, because they then will be added to DEPENDS (as I said IMO bad idea) and have to remove those from DEPENDS (DEPENDS_remove) and add the correct recipe name manually. regards, Claudius > > [pkg-rels]: https://www.debian.org/doc/debian-policy/ch-relationships.html "Debian policy: Declaring relationships between packages" > [pkg-control]: https://www.debian.org/doc/debian-policy/ch-controlfields.html "Debian policy: Control files and their fields" > [dpkg-raw]: https://github.com/ilbers/isar/blob/6c5db020b9b837d7b0ce63bfc719f9192e725f26/meta/classes/dpkg-raw.bbclass > [bb-tasks]: https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html#executing-tasks "Bitbake: Executing Tasks" > [bb-rdepends]: https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html#var-RDEPENDS "Bitbake: RDEPENDS" > [bb-rrecommends]: https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html#var-RRECOMMENDS "Bitbake: RRECOMMENDS" > [bb-packages]: https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html#var-PACKAGES "Bitbake: PACKAGES" > -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de