Re: [PATCH 1/1] meta: added do_transform_template task as templating system and switch

[Jan Kiszka <jan.kiszka@siemens.com>]

On 04.02.19 14:52, [ext] claudius.heine.ext@siemens.com wrote:
> From: Claudius Heine <ch@denx.de>
> 
> The new introduced variables are TEMPLATE_FILES and TEMPLATE_VARS.
> TEMPLATE_FILES is a list of files that should be processed in the
> `do_transform_template` task and TEMPLATE_VARS is a list of bitbake
> variable names that should be substituted in the template file.
> 
> Signed-off-by: Claudius Heine <ch@denx.de>
> ---
>   doc/technical_overview.md                     | 25 ++++++++
>   meta/classes/base.bbclass                     |  1 +
>   meta/classes/dpkg-base.bbclass                |  2 +-
>   meta/classes/template.bbclass                 | 62 +++++++++++++++++++
>   .../debian/{changelog => changelog.tmpl}      |  2 +-
>   meta/recipes-bsp/u-boot/files/debian/control  | 19 ------
>   .../u-boot/files/debian/control.tmpl          | 19 ++++++
>   meta/recipes-bsp/u-boot/u-boot-custom.inc     | 12 ++--
>   .../debian/{changelog => changelog.tmpl}      |  2 +-
>   .../linux-module/files/debian/control         | 11 ----
>   .../linux-module/files/debian/control.tmpl    | 11 ++++
>   meta/recipes-kernel/linux-module/module.inc   |  7 +--
>   12 files changed, 129 insertions(+), 44 deletions(-)
>   create mode 100644 meta/classes/template.bbclass
>   rename meta/recipes-bsp/u-boot/files/debian/{changelog => changelog.tmpl} (74%)
>   delete mode 100644 meta/recipes-bsp/u-boot/files/debian/control
>   create mode 100644 meta/recipes-bsp/u-boot/files/debian/control.tmpl
>   rename meta/recipes-kernel/linux-module/files/debian/{changelog => changelog.tmpl} (74%)
>   delete mode 100644 meta/recipes-kernel/linux-module/files/debian/control
>   create mode 100644 meta/recipes-kernel/linux-module/files/debian/control.tmpl
> 
> diff --git a/doc/technical_overview.md b/doc/technical_overview.md
> index 7efbeee..2608727 100644
> --- a/doc/technical_overview.md
> +++ b/doc/technical_overview.md
> @@ -283,3 +283,28 @@ cross-compilation, this is done by the same tools as for native compilation.
>   Depending on ISAR_CROSS_COMPILE value, additional architecture specifiers are
>   passed to build tools automatically, so this is absolutely transparent from
>   the user point of view.
> +
> +## 3.9 Additional features
> +
> +### 3.9.1 Template files
> +
> +A basic templating system implemented on top `envsubst(1)` is available. It
> +allows limited access to bitbake variables for file generation.
> +
> +This system is implemented in the `template.bbclass` and defines the
> +`do_transform_template` task, the `TEMPLATE_FILES` and `TEMPLATE_VARS`
> +variables. Tasks that use need to use files generated via templates need
> +to be executed after the `do_transform_template` task.
> +
> +The `TEMPLATE_FILES` variable contains a space seperated list of template

separated, also below

> +files and the `TEMPLATE_VARS` variable a space seperated list of bitbake
> +variable names. The `do_transform_template` task takes these and generates
> +files using `envsubst(1)`. The output files are placed in the same directory
> +as the template files. The name of the output files are eighter the name of the

either

(maybe turn your spell checker on in your editor ;) )

> +template file with `.tmpl` removed from the file name end or, if the template
> +file did not end with `.tmpl`, the name of the template file with `.out` added
> +to the end.

I wonder if we shouldn't use '.in' instead of '.tmpl'.

> +
> +Only template files from the `WORKDIR` are accepted. Eighter specify relative
> +paths based on the recipes `WORKDIR` or absolute paths containing the `WORKDIR`
> +in the `TEMPLATE_FILES` variable.
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index d4082de..5ab6fa4 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -25,6 +25,7 @@ die() {
>   }
>   
>   inherit logging
> +inherit template
>   
>   # Derived from bitbake: bitbake/classes/base.bbclass
>   addtask showdata
> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
> index f1b127c..fe96850 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -40,7 +40,7 @@ do_prepare_build() {
>       true
>   }
>   
> -addtask prepare_build after do_patch before do_build
> +addtask prepare_build after do_patch do_transform_template before do_build
>   do_prepare_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>   # If Isar recipes depend on each other, they typically need the package
>   # deployed to isar-apt
> diff --git a/meta/classes/template.bbclass b/meta/classes/template.bbclass
> new file mode 100644
> index 0000000..716b7a5
> --- /dev/null
> +++ b/meta/classes/template.bbclass
> @@ -0,0 +1,62 @@
> +# This software is a part of ISAR.
> +# Copyright (C) Siemens AG, 2019
> +#
> +# SPDX-License-Identifier: MIT
> +
> +TEMPLATE_FILES ??= ""

Needed? You handle the empty case I think.

> +TEMPLATE_VARS ??= "PN PV DESCRIPTION HOMEPAGE MAINTAINER KERNEL_NAME MACHINE \
> +                   DISTRO_ARCH"

I don't think MAINTAINER qualifies. DESCRIPTION and MAINTAINER have both /some/ 
overlap with recipes (though both only by convention these days, it seems), but 
MAINTAINER is a Debian-only thing that goes into control files directly (usually 
with dummy values).

We should probably pull this list into out bitbake.conf, rather than encoding it 
in the implementation of the class.

> +
> +do_transform_template[vardeps] = "TEMPLATE_FILES ${TEMPLATE_VARS}"
> +do_transform_template[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> +python do_transform_template() {
> +    import subprocess, contextlib
> +
> +    workdir = os.path.normpath(d.getVar('WORKDIR', True))
> +
> +    template_vars = (d.getVar('TEMPLATE_VARS', True) or "").split()
> +    if len(template_vars) == 0:
> +        return
> +
> +    template_files = (d.getVar('TEMPLATE_FILES', True) or "").split()
> +    if len(template_files) == 0:
> +        return
> +
> +    cmd = "envsubst"
> +    args = " ".join("\${{{}}}".format(i) for i in template_vars)
> +
> +    # Copy current process environment and add template variables
> +    # from bitbake data store:
> +    env = os.environ.copy()
> +    for varname in template_vars:
> +        value = d.getVar(varname, True)
> +        if value:
> +            env.update({varname: value})
> +
> +    for template_file in template_files:
> +        # Normpath and workdir checks should prevent accidential or
> +        # uninformed changes to files outside of the tmp and workdirectoy
> +        template_file = os.path.normpath(template_file)
> +
> +        # Convert relative paths to absolut paths based on the workdir:
> +        if not os.path.isabs(template_file):
> +            template_file = os.path.normpath(os.path.join(workdir, template_file))
> +
> +        if not template_file.startswith(workdir):
> +            bb.fatal("Template file ({}) is not within workdir ({})"
> +                     .format(template_file, workdir))
> +
> +        output_file = (os.path.splitext(template_file)[0]
> +                       if template_file.endswith(".tmpl")
> +                       else (template_file + ".out"))
> +        bb.note("{} {} [in: {} out: {}]".format(cmd, args,
> +                                                template_file, output_file))
> +        with contextlib.ExitStack() as stack:
> +            input = stack.enter_context(open(template_file, 'rb'))
> +            output = stack.enter_context(open(output_file, 'wb'))
> +            process = subprocess.Popen([cmd, args], stdin=input,
> +                                       stdout=output, env=env)
> +            if process.wait() != 0:
> +                bb.fatal("processing of template failed")
> +}
> +addtask do_transform_template after do_unpack before do_build
> diff --git a/meta/recipes-bsp/u-boot/files/debian/changelog b/meta/recipes-bsp/u-boot/files/debian/changelog.tmpl
> similarity index 74%
> rename from meta/recipes-bsp/u-boot/files/debian/changelog
> rename to meta/recipes-bsp/u-boot/files/debian/changelog.tmpl
> index c1c3516..6e59e06 100644
> --- a/meta/recipes-bsp/u-boot/files/debian/changelog
> +++ b/meta/recipes-bsp/u-boot/files/debian/changelog.tmpl
> @@ -1,4 +1,4 @@
> -@PN@ (@PV@) unstable; urgency=low
> +${PN} (${PV}) unstable; urgency=low
>   
>     * Generated package.
>   
> diff --git a/meta/recipes-bsp/u-boot/files/debian/control b/meta/recipes-bsp/u-boot/files/debian/control
> deleted file mode 100644
> index 6b4c839..0000000
> --- a/meta/recipes-bsp/u-boot/files/debian/control
> +++ /dev/null
> @@ -1,19 +0,0 @@
> -Source: @PN@
> -Section: admin
> -Priority: optional
> -Standards-Version: 3.9.6
> -Build-Depends: @BUILD_DEPENDS@
> -Maintainer: ISAR project <isar-users@googlegroups.com>
> -
> -Package: u-boot-@MACHINE@
> -Architecture: @DISTRO_ARCH@
> -Description: @DESCRIPTION@, bootloader binaries
> -
> -Package: u-boot-@MACHINE@-dev
> -Architecture: @DISTRO_ARCH@
> -Description: @DESCRIPTION@, bootloader libraries
> -
> -Package: u-boot-tools
> -Architecture: linux-any
> -Depends: ${shlibs:Depends}, ${misc:Depends}
> -Description: @DESCRIPTION@, companion tools
> diff --git a/meta/recipes-bsp/u-boot/files/debian/control.tmpl b/meta/recipes-bsp/u-boot/files/debian/control.tmpl
> new file mode 100644
> index 0000000..5c1cc92
> --- /dev/null
> +++ b/meta/recipes-bsp/u-boot/files/debian/control.tmpl
> @@ -0,0 +1,19 @@
> +Source: ${PN}
> +Section: admin
> +Priority: optional
> +Standards-Version: 3.9.6
> +Build-Depends: ${BUILD_DEPENDS}
> +Maintainer: ISAR project <isar-users@googlegroups.com>
> +
> +Package: u-boot-${MACHINE}
> +Architecture: ${DISTRO_ARCH}
> +Description: ${DESCRIPTION}, bootloader binaries
> +
> +Package: u-boot-${MACHINE}-dev
> +Architecture: ${DISTRO_ARCH}
> +Description: ${DESCRIPTION}, bootloader libraries
> +
> +Package: u-boot-tools
> +Architecture: linux-any
> +Depends: ${shlibs:Depends}, ${misc:Depends}
> +Description: ${DESCRIPTION}, companion tools
> diff --git a/meta/recipes-bsp/u-boot/u-boot-custom.inc b/meta/recipes-bsp/u-boot/u-boot-custom.inc
> index 4b38c88..c6aab43 100644
> --- a/meta/recipes-bsp/u-boot/u-boot-custom.inc
> +++ b/meta/recipes-bsp/u-boot/u-boot-custom.inc
> @@ -15,16 +15,14 @@ inherit dpkg
>   
>   SRC_URI += "file://debian/"
>   
> -U_BOOT_BUILD_DEPENDS ?= "bc, bison, flex, device-tree-compiler"
> +BUILD_DEPENDS ?= "bc, bison, flex, device-tree-compiler"
> +
> +TEMPLATE_FILES = "debian/changelog.tmpl \
> +                  debian/control.tmpl"
> +TEMPLATE_VARS += "BUILD_DEPENDS"
>   
>   do_prepare_build() {
>       cp -r ${WORKDIR}/debian ${S}/
> -    sed -i -e 's/@PN@/${PN}/g' -e 's/@PV@/${PV}/g' \
> -        -e 's/@BUILD_DEPENDS@/${U_BOOT_BUILD_DEPENDS}/g' \
> -        -e 's/@MACHINE@/${MACHINE}/g' \
> -        -e 's/@DISTRO_ARCH@/${DISTRO_ARCH}/g' \
> -        -e 's/@DESCRIPTION@/${DESCRIPTION}/g' \
> -        ${S}/debian/changelog ${S}/debian/control
>   
>       echo "${U_BOOT_BIN} /usr/lib/u-boot/${MACHINE}" > \
>           ${S}/debian/u-boot-${MACHINE}.install
> diff --git a/meta/recipes-kernel/linux-module/files/debian/changelog b/meta/recipes-kernel/linux-module/files/debian/changelog.tmpl
> similarity index 74%
> rename from meta/recipes-kernel/linux-module/files/debian/changelog
> rename to meta/recipes-kernel/linux-module/files/debian/changelog.tmpl
> index c1c3516..6e59e06 100644
> --- a/meta/recipes-kernel/linux-module/files/debian/changelog
> +++ b/meta/recipes-kernel/linux-module/files/debian/changelog.tmpl
> @@ -1,4 +1,4 @@
> -@PN@ (@PV@) unstable; urgency=low
> +${PN} (${PV}) unstable; urgency=low
>   
>     * Generated package.
>   
> diff --git a/meta/recipes-kernel/linux-module/files/debian/control b/meta/recipes-kernel/linux-module/files/debian/control
> deleted file mode 100644
> index 1ee634c..0000000
> --- a/meta/recipes-kernel/linux-module/files/debian/control
> +++ /dev/null
> @@ -1,11 +0,0 @@
> -Source: @PN@
> -Section: kernel
> -Priority: optional
> -Standards-Version: 3.9.6
> -Build-Depends: linux-headers-@KERNEL_NAME@
> -Maintainer: ISAR project <isar-users@googlegroups.com>
> -
> -Package: @PN@
> -Architecture: any
> -Depends: linux-image-@KERNEL_NAME@, kmod
> -Description: @DESCRIPTION@
> diff --git a/meta/recipes-kernel/linux-module/files/debian/control.tmpl b/meta/recipes-kernel/linux-module/files/debian/control.tmpl
> new file mode 100644
> index 0000000..3b3292d
> --- /dev/null
> +++ b/meta/recipes-kernel/linux-module/files/debian/control.tmpl
> @@ -0,0 +1,11 @@
> +Source: ${PN}
> +Section: kernel
> +Priority: optional
> +Standards-Version: 3.9.6
> +Build-Depends: linux-headers-${KERNEL_NAME}
> +Maintainer: ISAR project <isar-users@googlegroups.com>
> +
> +Package: ${PN}
> +Architecture: any
> +Depends: linux-image-${KERNEL_NAME}, kmod
> +Description: ${DESCRIPTION}
> diff --git a/meta/recipes-kernel/linux-module/module.inc b/meta/recipes-kernel/linux-module/module.inc
> index cb7b8ad..44edd58 100644
> --- a/meta/recipes-kernel/linux-module/module.inc
> +++ b/meta/recipes-kernel/linux-module/module.inc
> @@ -19,12 +19,11 @@ AUTOLOAD ?= ""
>   
>   inherit dpkg
>   
> +TEMPLATE_FILES = "debian/control.tmpl \
> +                  debian/changelog.tmpl"
> +
>   do_prepare_build() {
>       cp -r ${WORKDIR}/debian ${S}/
> -    sed -i -e 's/@PN@/${PN}/g' -e 's/@PV@/${PV}/g' \
> -           -e 's/@KERNEL_NAME@/${KERNEL_NAME}/g' \
> -           -e 's/@DESCRIPTION@/${DESCRIPTION}/g' \
> -        ${S}/debian/changelog ${S}/debian/control
>   
>       for module in "${AUTOLOAD}"; do
>           echo "echo $module >> /etc/modules" >> ${S}/debian/postinst
> 

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux