Re: [PATCH v2 1/1] meta: added do_transform_template task as templating system and switch

[Henning Schild <henning.schild@siemens.com>]

Looks good and useful to me. I did not try it but since the
example-module is a user of it we do have CI coverage.

Henning

Am Tue, 5 Feb 2019 12:18:01 +0100
schrieb "[ext] claudius.heine.ext@siemens.com"
<claudius.heine.ext@siemens.com>:

> From: Claudius Heine <ch@denx.de>
> 
> The new introduced variables are TEMPLATE_FILES and TEMPLATE_VARS.
> TEMPLATE_FILES is a list of files that should be processed in the
> `do_transform_template` task and TEMPLATE_VARS is a list of bitbake
> variable names that should be substituted in the template file.
> 
> Signed-off-by: Claudius Heine <ch@denx.de>
> ---
>  doc/technical_overview.md                     | 25 ++++++++
>  meta/classes/base.bbclass                     |  1 +
>  meta/classes/dpkg-base.bbclass                |  2 +-
>  meta/classes/template.bbclass                 | 62
> +++++++++++++++++++ .../debian/{changelog => changelog.tmpl}      |
> 2 +- meta/recipes-bsp/u-boot/files/debian/control  | 19 ------
>  .../u-boot/files/debian/control.tmpl          | 19 ++++++
>  meta/recipes-bsp/u-boot/u-boot-custom.inc     | 12 ++--
>  .../debian/{changelog => changelog.tmpl}      |  2 +-
>  .../linux-module/files/debian/control         | 11 ----
>  .../linux-module/files/debian/control.tmpl    | 11 ++++
>  meta/recipes-kernel/linux-module/module.inc   |  7 +--
>  12 files changed, 129 insertions(+), 44 deletions(-)
>  create mode 100644 meta/classes/template.bbclass
>  rename meta/recipes-bsp/u-boot/files/debian/{changelog =>
> changelog.tmpl} (74%) delete mode 100644
> meta/recipes-bsp/u-boot/files/debian/control create mode 100644
> meta/recipes-bsp/u-boot/files/debian/control.tmpl rename
> meta/recipes-kernel/linux-module/files/debian/{changelog =>
> changelog.tmpl} (74%) delete mode 100644
> meta/recipes-kernel/linux-module/files/debian/control create mode
> 100644 meta/recipes-kernel/linux-module/files/debian/control.tmpl
> 
> diff --git a/doc/technical_overview.md b/doc/technical_overview.md
> index 7efbeee..0ddcabc 100644
> --- a/doc/technical_overview.md
> +++ b/doc/technical_overview.md
> @@ -283,3 +283,28 @@ cross-compilation, this is done by the same
> tools as for native compilation. Depending on ISAR_CROSS_COMPILE
> value, additional architecture specifiers are passed to build tools
> automatically, so this is absolutely transparent from the user point
> of view. +
> +## 3.9 Additional features
> +
> +### 3.9.1 Template files
> +
> +A basic templating system implemented on top `envsubst(1)` is
> available. It +allows limited access to bitbake variables for file
> generation. +
> +This system is implemented in the `template.bbclass` and defines the
> +`do_transform_template` task, the `TEMPLATE_FILES` and
> `TEMPLATE_VARS` +variables. Tasks that use need to use files
> generated via templates need +to be executed after the
> `do_transform_template` task. +
> +The `TEMPLATE_FILES` variable contains a space separated list of
> template +files and the `TEMPLATE_VARS` variable a space separated
> list of bitbake +variable names. The `do_transform_template` task
> takes these and generates +files using `envsubst(1)`. The output
> files are placed in the same directory +as the template files. The
> name of the output files are either the name of the +template file
> with `.tmpl` removed from the file name end or, if the template +file
> did not end with `.tmpl`, the name of the template file with `.out`
> added +to the end. +
> +Only template files from the `WORKDIR` are accepted. Either specify
> relative +paths based on the recipes `WORKDIR` or absolute paths
> containing the `WORKDIR` +in the `TEMPLATE_FILES` variable.
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index d4082de..5ab6fa4 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -25,6 +25,7 @@ die() {
>  }
>  
>  inherit logging
> +inherit template
>  
>  # Derived from bitbake: bitbake/classes/base.bbclass
>  addtask showdata
> diff --git a/meta/classes/dpkg-base.bbclass
> b/meta/classes/dpkg-base.bbclass index f1b127c..fe96850 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -40,7 +40,7 @@ do_prepare_build() {
>      true
>  }
>  
> -addtask prepare_build after do_patch before do_build
> +addtask prepare_build after do_patch do_transform_template before
> do_build do_prepare_build[stamp-extra-info] =
> "${DISTRO}-${DISTRO_ARCH}" # If Isar recipes depend on each other,
> they typically need the package # deployed to isar-apt
> diff --git a/meta/classes/template.bbclass
> b/meta/classes/template.bbclass new file mode 100644
> index 0000000..716b7a5
> --- /dev/null
> +++ b/meta/classes/template.bbclass
> @@ -0,0 +1,62 @@
> +# This software is a part of ISAR.
> +# Copyright (C) Siemens AG, 2019
> +#
> +# SPDX-License-Identifier: MIT
> +
> +TEMPLATE_FILES ??= ""
> +TEMPLATE_VARS ??= "PN PV DESCRIPTION HOMEPAGE MAINTAINER KERNEL_NAME
> MACHINE \
> +                   DISTRO_ARCH"
> +
> +do_transform_template[vardeps] = "TEMPLATE_FILES ${TEMPLATE_VARS}"
> +do_transform_template[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> +python do_transform_template() {
> +    import subprocess, contextlib
> +
> +    workdir = os.path.normpath(d.getVar('WORKDIR', True))
> +
> +    template_vars = (d.getVar('TEMPLATE_VARS', True) or "").split()
> +    if len(template_vars) == 0:
> +        return
> +
> +    template_files = (d.getVar('TEMPLATE_FILES', True) or "").split()
> +    if len(template_files) == 0:
> +        return
> +
> +    cmd = "envsubst"
> +    args = " ".join("\${{{}}}".format(i) for i in template_vars)
> +
> +    # Copy current process environment and add template variables
> +    # from bitbake data store:
> +    env = os.environ.copy()
> +    for varname in template_vars:
> +        value = d.getVar(varname, True)
> +        if value:
> +            env.update({varname: value})
> +
> +    for template_file in template_files:
> +        # Normpath and workdir checks should prevent accidential or
> +        # uninformed changes to files outside of the tmp and
> workdirectoy
> +        template_file = os.path.normpath(template_file)
> +
> +        # Convert relative paths to absolut paths based on the
> workdir:
> +        if not os.path.isabs(template_file):
> +            template_file = os.path.normpath(os.path.join(workdir,
> template_file)) +
> +        if not template_file.startswith(workdir):
> +            bb.fatal("Template file ({}) is not within workdir ({})"
> +                     .format(template_file, workdir))
> +
> +        output_file = (os.path.splitext(template_file)[0]
> +                       if template_file.endswith(".tmpl")
> +                       else (template_file + ".out"))
> +        bb.note("{} {} [in: {} out: {}]".format(cmd, args,
> +                                                template_file,
> output_file))
> +        with contextlib.ExitStack() as stack:
> +            input = stack.enter_context(open(template_file, 'rb'))
> +            output = stack.enter_context(open(output_file, 'wb'))
> +            process = subprocess.Popen([cmd, args], stdin=input,
> +                                       stdout=output, env=env)
> +            if process.wait() != 0:
> +                bb.fatal("processing of template failed")
> +}
> +addtask do_transform_template after do_unpack before do_build
> diff --git a/meta/recipes-bsp/u-boot/files/debian/changelog
> b/meta/recipes-bsp/u-boot/files/debian/changelog.tmpl similarity
> index 74% rename from meta/recipes-bsp/u-boot/files/debian/changelog
> rename to meta/recipes-bsp/u-boot/files/debian/changelog.tmpl
> index c1c3516..6e59e06 100644
> --- a/meta/recipes-bsp/u-boot/files/debian/changelog
> +++ b/meta/recipes-bsp/u-boot/files/debian/changelog.tmpl
> @@ -1,4 +1,4 @@
> -@PN@ (@PV@) unstable; urgency=low
> +${PN} (${PV}) unstable; urgency=low
>  
>    * Generated package.
>  
> diff --git a/meta/recipes-bsp/u-boot/files/debian/control
> b/meta/recipes-bsp/u-boot/files/debian/control deleted file mode
> 100644 index 6b4c839..0000000
> --- a/meta/recipes-bsp/u-boot/files/debian/control
> +++ /dev/null
> @@ -1,19 +0,0 @@
> -Source: @PN@
> -Section: admin
> -Priority: optional
> -Standards-Version: 3.9.6
> -Build-Depends: @BUILD_DEPENDS@
> -Maintainer: ISAR project <isar-users@googlegroups.com>
> -
> -Package: u-boot-@MACHINE@
> -Architecture: @DISTRO_ARCH@
> -Description: @DESCRIPTION@, bootloader binaries
> -
> -Package: u-boot-@MACHINE@-dev
> -Architecture: @DISTRO_ARCH@
> -Description: @DESCRIPTION@, bootloader libraries
> -
> -Package: u-boot-tools
> -Architecture: linux-any
> -Depends: ${shlibs:Depends}, ${misc:Depends}
> -Description: @DESCRIPTION@, companion tools
> diff --git a/meta/recipes-bsp/u-boot/files/debian/control.tmpl
> b/meta/recipes-bsp/u-boot/files/debian/control.tmpl new file mode
> 100644 index 0000000..5c1cc92
> --- /dev/null
> +++ b/meta/recipes-bsp/u-boot/files/debian/control.tmpl
> @@ -0,0 +1,19 @@
> +Source: ${PN}
> +Section: admin
> +Priority: optional
> +Standards-Version: 3.9.6
> +Build-Depends: ${BUILD_DEPENDS}
> +Maintainer: ISAR project <isar-users@googlegroups.com>
> +
> +Package: u-boot-${MACHINE}
> +Architecture: ${DISTRO_ARCH}
> +Description: ${DESCRIPTION}, bootloader binaries
> +
> +Package: u-boot-${MACHINE}-dev
> +Architecture: ${DISTRO_ARCH}
> +Description: ${DESCRIPTION}, bootloader libraries
> +
> +Package: u-boot-tools
> +Architecture: linux-any
> +Depends: ${shlibs:Depends}, ${misc:Depends}
> +Description: ${DESCRIPTION}, companion tools
> diff --git a/meta/recipes-bsp/u-boot/u-boot-custom.inc
> b/meta/recipes-bsp/u-boot/u-boot-custom.inc index 4b38c88..c6aab43
> 100644 --- a/meta/recipes-bsp/u-boot/u-boot-custom.inc
> +++ b/meta/recipes-bsp/u-boot/u-boot-custom.inc
> @@ -15,16 +15,14 @@ inherit dpkg
>  
>  SRC_URI += "file://debian/"
>  
> -U_BOOT_BUILD_DEPENDS ?= "bc, bison, flex, device-tree-compiler"
> +BUILD_DEPENDS ?= "bc, bison, flex, device-tree-compiler"
> +
> +TEMPLATE_FILES = "debian/changelog.tmpl \
> +                  debian/control.tmpl"
> +TEMPLATE_VARS += "BUILD_DEPENDS"
>  
>  do_prepare_build() {
>      cp -r ${WORKDIR}/debian ${S}/
> -    sed -i -e 's/@PN@/${PN}/g' -e 's/@PV@/${PV}/g' \
> -        -e 's/@BUILD_DEPENDS@/${U_BOOT_BUILD_DEPENDS}/g' \
> -        -e 's/@MACHINE@/${MACHINE}/g' \
> -        -e 's/@DISTRO_ARCH@/${DISTRO_ARCH}/g' \
> -        -e 's/@DESCRIPTION@/${DESCRIPTION}/g' \
> -        ${S}/debian/changelog ${S}/debian/control
>  
>      echo "${U_BOOT_BIN} /usr/lib/u-boot/${MACHINE}" > \
>          ${S}/debian/u-boot-${MACHINE}.install
> diff --git a/meta/recipes-kernel/linux-module/files/debian/changelog
> b/meta/recipes-kernel/linux-module/files/debian/changelog.tmpl
> similarity index 74% rename from
> meta/recipes-kernel/linux-module/files/debian/changelog rename to
> meta/recipes-kernel/linux-module/files/debian/changelog.tmpl index
> c1c3516..6e59e06 100644 ---
> a/meta/recipes-kernel/linux-module/files/debian/changelog +++
> b/meta/recipes-kernel/linux-module/files/debian/changelog.tmpl @@
> -1,4 +1,4 @@ -@PN@ (@PV@) unstable; urgency=low
> +${PN} (${PV}) unstable; urgency=low
>  
>    * Generated package.
>  
> diff --git a/meta/recipes-kernel/linux-module/files/debian/control
> b/meta/recipes-kernel/linux-module/files/debian/control deleted file
> mode 100644 index 1ee634c..0000000
> --- a/meta/recipes-kernel/linux-module/files/debian/control
> +++ /dev/null
> @@ -1,11 +0,0 @@
> -Source: @PN@
> -Section: kernel
> -Priority: optional
> -Standards-Version: 3.9.6
> -Build-Depends: linux-headers-@KERNEL_NAME@
> -Maintainer: ISAR project <isar-users@googlegroups.com>
> -
> -Package: @PN@
> -Architecture: any
> -Depends: linux-image-@KERNEL_NAME@, kmod
> -Description: @DESCRIPTION@
> diff --git
> a/meta/recipes-kernel/linux-module/files/debian/control.tmpl
> b/meta/recipes-kernel/linux-module/files/debian/control.tmpl new file
> mode 100644 index 0000000..3b3292d --- /dev/null
> +++ b/meta/recipes-kernel/linux-module/files/debian/control.tmpl
> @@ -0,0 +1,11 @@
> +Source: ${PN}
> +Section: kernel
> +Priority: optional
> +Standards-Version: 3.9.6
> +Build-Depends: linux-headers-${KERNEL_NAME}
> +Maintainer: ISAR project <isar-users@googlegroups.com>
> +
> +Package: ${PN}
> +Architecture: any
> +Depends: linux-image-${KERNEL_NAME}, kmod
> +Description: ${DESCRIPTION}
> diff --git a/meta/recipes-kernel/linux-module/module.inc
> b/meta/recipes-kernel/linux-module/module.inc index cb7b8ad..44edd58
> 100644 --- a/meta/recipes-kernel/linux-module/module.inc
> +++ b/meta/recipes-kernel/linux-module/module.inc
> @@ -19,12 +19,11 @@ AUTOLOAD ?= ""
>  
>  inherit dpkg
>  
> +TEMPLATE_FILES = "debian/control.tmpl \
> +                  debian/changelog.tmpl"
> +
>  do_prepare_build() {
>      cp -r ${WORKDIR}/debian ${S}/
> -    sed -i -e 's/@PN@/${PN}/g' -e 's/@PV@/${PV}/g' \
> -           -e 's/@KERNEL_NAME@/${KERNEL_NAME}/g' \
> -           -e 's/@DESCRIPTION@/${DESCRIPTION}/g' \
> -        ${S}/debian/changelog ${S}/debian/control
>  
>      for module in "${AUTOLOAD}"; do
>          echo "echo $module >> /etc/modules" >> ${S}/debian/postinst