From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6652594697292218368 X-Received: by 2002:a2e:7803:: with SMTP id t3-v6mr43654ljc.11.1549302915882; Mon, 04 Feb 2019 09:55:15 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a19:ca5d:: with SMTP id h29ls13611lfj.12.gmail; Mon, 04 Feb 2019 09:55:15 -0800 (PST) X-Google-Smtp-Source: AHgI3IYDuBR3jq17Q8/Tzpz8gH6qQLZjCJ6xTeXo3PEAT74aic6vrEocrdkOPrpLg0kpWFtpyH/6 X-Received: by 2002:a19:fc11:: with SMTP id a17mr44369lfi.11.1549302915049; Mon, 04 Feb 2019 09:55:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1549302915; cv=none; d=google.com; s=arc-20160816; b=jBu/PUJcNKD4HZacvL/MVp+Rd9zWWSYD1KmjvXT0+188NOzBfSvj/Wk4vVLS6pZw55 oKn8xT29geiOcv0OfmoPu3tPP01IMgsOkSaJy0fk0f/1bR1vCo79OZIqXitcdosPjs9x 4VJYymtxxRQOhs/BBCc2b06Ja3AYcS9qpvkTHMJlkZTSxtEwDiKjC6Sh1jZC+Brq5vXl NvVYZO+fKwM3CFUvXTeyCkGZHpRMWasrG9+Zwqlvxszt45eAf6lGTo1/L7/BGkAI063e xRwNE6y3WLeNnxydtrxLkA9w1Ph0X5BzKZKzh+CINBwRzKLnB598E7uGW3XJ5PkK01kW pAHw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject; bh=VH5a6O4qdN+DjDLqBqzfiPKF1xhd7A5Fi1IdmLnaPcA=; b=FRhb+0p3NsXDl5+AvCMR2kj6O9OImFVad5ddyxLWF6TmU0ZkgNz+VmRDth85MfxiEA tqN/G4OQPecZczYO/gHK1dAdXYqXMKoK6CbhQQ4dV9pGbAq2gc5kUeCLGXVRPNZ+AFbu +ldxsqvAtlOH4A/X9GONDlQb9Ev+mWX2WvM60h0iwzpbcLERtWgKiCMAsn4YcY1Ch+D2 JV3L1znqxN6T6gwfJXwrt7bHlCGl1oHZdoC85SkmNd9gPWvsqePptu3am1LoaK4wx8xA 2H4eOKdTiru2tiH5W1rR4pX40ImG2/aaScL/Z0Mx110GAi+fbsjunRdKJVTlzpPSR7+S 33Ng== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.40 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Return-Path: Received: from gecko.sbs.de (gecko.sbs.de. [194.138.37.40]) by gmr-mx.google.com with ESMTPS id h11si377312lfm.1.2019.02.04.09.55.14 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 04 Feb 2019 09:55:14 -0800 (PST) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.40 as permitted sender) client-ip=194.138.37.40; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.40 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by gecko.sbs.de (8.15.2/8.15.2) with ESMTPS id x14HtD4r001332 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 4 Feb 2019 18:55:14 +0100 Received: from [139.25.68.37] (md1q0hnc.ad001.siemens.net [139.25.68.37] (may be forged)) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id x14HtDNr003111; Mon, 4 Feb 2019 18:55:13 +0100 Subject: Re: [PATCH 1/1] meta: added do_transform_template task as templating system and switch To: "[ext] claudius.heine.ext@siemens.com" , isar-users@googlegroups.com Cc: Claudius Heine References: <20190204135241.32000-1-claudius.heine.ext@siemens.com> <20190204135241.32000-2-claudius.heine.ext@siemens.com> From: Jan Kiszka Message-ID: Date: Mon, 4 Feb 2019 18:55:12 +0100 User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.8.1.12) Gecko/20080226 SUSE/2.0.0.12-1.1 Thunderbird/2.0.0.12 Mnenhy/0.7.5.666 MIME-Version: 1.0 In-Reply-To: <20190204135241.32000-2-claudius.heine.ext@siemens.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: nY7wJIa5rEAY On 04.02.19 14:52, [ext] claudius.heine.ext@siemens.com wrote: > From: Claudius Heine > > The new introduced variables are TEMPLATE_FILES and TEMPLATE_VARS. > TEMPLATE_FILES is a list of files that should be processed in the > `do_transform_template` task and TEMPLATE_VARS is a list of bitbake > variable names that should be substituted in the template file. > > Signed-off-by: Claudius Heine > --- > doc/technical_overview.md | 25 ++++++++ > meta/classes/base.bbclass | 1 + > meta/classes/dpkg-base.bbclass | 2 +- > meta/classes/template.bbclass | 62 +++++++++++++++++++ > .../debian/{changelog => changelog.tmpl} | 2 +- > meta/recipes-bsp/u-boot/files/debian/control | 19 ------ > .../u-boot/files/debian/control.tmpl | 19 ++++++ > meta/recipes-bsp/u-boot/u-boot-custom.inc | 12 ++-- > .../debian/{changelog => changelog.tmpl} | 2 +- > .../linux-module/files/debian/control | 11 ---- > .../linux-module/files/debian/control.tmpl | 11 ++++ > meta/recipes-kernel/linux-module/module.inc | 7 +-- > 12 files changed, 129 insertions(+), 44 deletions(-) > create mode 100644 meta/classes/template.bbclass > rename meta/recipes-bsp/u-boot/files/debian/{changelog => changelog.tmpl} (74%) > delete mode 100644 meta/recipes-bsp/u-boot/files/debian/control > create mode 100644 meta/recipes-bsp/u-boot/files/debian/control.tmpl > rename meta/recipes-kernel/linux-module/files/debian/{changelog => changelog.tmpl} (74%) > delete mode 100644 meta/recipes-kernel/linux-module/files/debian/control > create mode 100644 meta/recipes-kernel/linux-module/files/debian/control.tmpl > > diff --git a/doc/technical_overview.md b/doc/technical_overview.md > index 7efbeee..2608727 100644 > --- a/doc/technical_overview.md > +++ b/doc/technical_overview.md > @@ -283,3 +283,28 @@ cross-compilation, this is done by the same tools as for native compilation. > Depending on ISAR_CROSS_COMPILE value, additional architecture specifiers are > passed to build tools automatically, so this is absolutely transparent from > the user point of view. > + > +## 3.9 Additional features > + > +### 3.9.1 Template files > + > +A basic templating system implemented on top `envsubst(1)` is available. It > +allows limited access to bitbake variables for file generation. > + > +This system is implemented in the `template.bbclass` and defines the > +`do_transform_template` task, the `TEMPLATE_FILES` and `TEMPLATE_VARS` > +variables. Tasks that use need to use files generated via templates need > +to be executed after the `do_transform_template` task. > + > +The `TEMPLATE_FILES` variable contains a space seperated list of template separated, also below > +files and the `TEMPLATE_VARS` variable a space seperated list of bitbake > +variable names. The `do_transform_template` task takes these and generates > +files using `envsubst(1)`. The output files are placed in the same directory > +as the template files. The name of the output files are eighter the name of the either (maybe turn your spell checker on in your editor ;) ) > +template file with `.tmpl` removed from the file name end or, if the template > +file did not end with `.tmpl`, the name of the template file with `.out` added > +to the end. I wonder if we shouldn't use '.in' instead of '.tmpl'. > + > +Only template files from the `WORKDIR` are accepted. Eighter specify relative > +paths based on the recipes `WORKDIR` or absolute paths containing the `WORKDIR` > +in the `TEMPLATE_FILES` variable. > diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass > index d4082de..5ab6fa4 100644 > --- a/meta/classes/base.bbclass > +++ b/meta/classes/base.bbclass > @@ -25,6 +25,7 @@ die() { > } > > inherit logging > +inherit template > > # Derived from bitbake: bitbake/classes/base.bbclass > addtask showdata > diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass > index f1b127c..fe96850 100644 > --- a/meta/classes/dpkg-base.bbclass > +++ b/meta/classes/dpkg-base.bbclass > @@ -40,7 +40,7 @@ do_prepare_build() { > true > } > > -addtask prepare_build after do_patch before do_build > +addtask prepare_build after do_patch do_transform_template before do_build > do_prepare_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" > # If Isar recipes depend on each other, they typically need the package > # deployed to isar-apt > diff --git a/meta/classes/template.bbclass b/meta/classes/template.bbclass > new file mode 100644 > index 0000000..716b7a5 > --- /dev/null > +++ b/meta/classes/template.bbclass > @@ -0,0 +1,62 @@ > +# This software is a part of ISAR. > +# Copyright (C) Siemens AG, 2019 > +# > +# SPDX-License-Identifier: MIT > + > +TEMPLATE_FILES ??= "" Needed? You handle the empty case I think. > +TEMPLATE_VARS ??= "PN PV DESCRIPTION HOMEPAGE MAINTAINER KERNEL_NAME MACHINE \ > + DISTRO_ARCH" I don't think MAINTAINER qualifies. DESCRIPTION and MAINTAINER have both /some/ overlap with recipes (though both only by convention these days, it seems), but MAINTAINER is a Debian-only thing that goes into control files directly (usually with dummy values). We should probably pull this list into out bitbake.conf, rather than encoding it in the implementation of the class. > + > +do_transform_template[vardeps] = "TEMPLATE_FILES ${TEMPLATE_VARS}" > +do_transform_template[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" > +python do_transform_template() { > + import subprocess, contextlib > + > + workdir = os.path.normpath(d.getVar('WORKDIR', True)) > + > + template_vars = (d.getVar('TEMPLATE_VARS', True) or "").split() > + if len(template_vars) == 0: > + return > + > + template_files = (d.getVar('TEMPLATE_FILES', True) or "").split() > + if len(template_files) == 0: > + return > + > + cmd = "envsubst" > + args = " ".join("\${{{}}}".format(i) for i in template_vars) > + > + # Copy current process environment and add template variables > + # from bitbake data store: > + env = os.environ.copy() > + for varname in template_vars: > + value = d.getVar(varname, True) > + if value: > + env.update({varname: value}) > + > + for template_file in template_files: > + # Normpath and workdir checks should prevent accidential or > + # uninformed changes to files outside of the tmp and workdirectoy > + template_file = os.path.normpath(template_file) > + > + # Convert relative paths to absolut paths based on the workdir: > + if not os.path.isabs(template_file): > + template_file = os.path.normpath(os.path.join(workdir, template_file)) > + > + if not template_file.startswith(workdir): > + bb.fatal("Template file ({}) is not within workdir ({})" > + .format(template_file, workdir)) > + > + output_file = (os.path.splitext(template_file)[0] > + if template_file.endswith(".tmpl") > + else (template_file + ".out")) > + bb.note("{} {} [in: {} out: {}]".format(cmd, args, > + template_file, output_file)) > + with contextlib.ExitStack() as stack: > + input = stack.enter_context(open(template_file, 'rb')) > + output = stack.enter_context(open(output_file, 'wb')) > + process = subprocess.Popen([cmd, args], stdin=input, > + stdout=output, env=env) > + if process.wait() != 0: > + bb.fatal("processing of template failed") > +} > +addtask do_transform_template after do_unpack before do_build > diff --git a/meta/recipes-bsp/u-boot/files/debian/changelog b/meta/recipes-bsp/u-boot/files/debian/changelog.tmpl > similarity index 74% > rename from meta/recipes-bsp/u-boot/files/debian/changelog > rename to meta/recipes-bsp/u-boot/files/debian/changelog.tmpl > index c1c3516..6e59e06 100644 > --- a/meta/recipes-bsp/u-boot/files/debian/changelog > +++ b/meta/recipes-bsp/u-boot/files/debian/changelog.tmpl > @@ -1,4 +1,4 @@ > -@PN@ (@PV@) unstable; urgency=low > +${PN} (${PV}) unstable; urgency=low > > * Generated package. > > diff --git a/meta/recipes-bsp/u-boot/files/debian/control b/meta/recipes-bsp/u-boot/files/debian/control > deleted file mode 100644 > index 6b4c839..0000000 > --- a/meta/recipes-bsp/u-boot/files/debian/control > +++ /dev/null > @@ -1,19 +0,0 @@ > -Source: @PN@ > -Section: admin > -Priority: optional > -Standards-Version: 3.9.6 > -Build-Depends: @BUILD_DEPENDS@ > -Maintainer: ISAR project > - > -Package: u-boot-@MACHINE@ > -Architecture: @DISTRO_ARCH@ > -Description: @DESCRIPTION@, bootloader binaries > - > -Package: u-boot-@MACHINE@-dev > -Architecture: @DISTRO_ARCH@ > -Description: @DESCRIPTION@, bootloader libraries > - > -Package: u-boot-tools > -Architecture: linux-any > -Depends: ${shlibs:Depends}, ${misc:Depends} > -Description: @DESCRIPTION@, companion tools > diff --git a/meta/recipes-bsp/u-boot/files/debian/control.tmpl b/meta/recipes-bsp/u-boot/files/debian/control.tmpl > new file mode 100644 > index 0000000..5c1cc92 > --- /dev/null > +++ b/meta/recipes-bsp/u-boot/files/debian/control.tmpl > @@ -0,0 +1,19 @@ > +Source: ${PN} > +Section: admin > +Priority: optional > +Standards-Version: 3.9.6 > +Build-Depends: ${BUILD_DEPENDS} > +Maintainer: ISAR project > + > +Package: u-boot-${MACHINE} > +Architecture: ${DISTRO_ARCH} > +Description: ${DESCRIPTION}, bootloader binaries > + > +Package: u-boot-${MACHINE}-dev > +Architecture: ${DISTRO_ARCH} > +Description: ${DESCRIPTION}, bootloader libraries > + > +Package: u-boot-tools > +Architecture: linux-any > +Depends: ${shlibs:Depends}, ${misc:Depends} > +Description: ${DESCRIPTION}, companion tools > diff --git a/meta/recipes-bsp/u-boot/u-boot-custom.inc b/meta/recipes-bsp/u-boot/u-boot-custom.inc > index 4b38c88..c6aab43 100644 > --- a/meta/recipes-bsp/u-boot/u-boot-custom.inc > +++ b/meta/recipes-bsp/u-boot/u-boot-custom.inc > @@ -15,16 +15,14 @@ inherit dpkg > > SRC_URI += "file://debian/" > > -U_BOOT_BUILD_DEPENDS ?= "bc, bison, flex, device-tree-compiler" > +BUILD_DEPENDS ?= "bc, bison, flex, device-tree-compiler" > + > +TEMPLATE_FILES = "debian/changelog.tmpl \ > + debian/control.tmpl" > +TEMPLATE_VARS += "BUILD_DEPENDS" > > do_prepare_build() { > cp -r ${WORKDIR}/debian ${S}/ > - sed -i -e 's/@PN@/${PN}/g' -e 's/@PV@/${PV}/g' \ > - -e 's/@BUILD_DEPENDS@/${U_BOOT_BUILD_DEPENDS}/g' \ > - -e 's/@MACHINE@/${MACHINE}/g' \ > - -e 's/@DISTRO_ARCH@/${DISTRO_ARCH}/g' \ > - -e 's/@DESCRIPTION@/${DESCRIPTION}/g' \ > - ${S}/debian/changelog ${S}/debian/control > > echo "${U_BOOT_BIN} /usr/lib/u-boot/${MACHINE}" > \ > ${S}/debian/u-boot-${MACHINE}.install > diff --git a/meta/recipes-kernel/linux-module/files/debian/changelog b/meta/recipes-kernel/linux-module/files/debian/changelog.tmpl > similarity index 74% > rename from meta/recipes-kernel/linux-module/files/debian/changelog > rename to meta/recipes-kernel/linux-module/files/debian/changelog.tmpl > index c1c3516..6e59e06 100644 > --- a/meta/recipes-kernel/linux-module/files/debian/changelog > +++ b/meta/recipes-kernel/linux-module/files/debian/changelog.tmpl > @@ -1,4 +1,4 @@ > -@PN@ (@PV@) unstable; urgency=low > +${PN} (${PV}) unstable; urgency=low > > * Generated package. > > diff --git a/meta/recipes-kernel/linux-module/files/debian/control b/meta/recipes-kernel/linux-module/files/debian/control > deleted file mode 100644 > index 1ee634c..0000000 > --- a/meta/recipes-kernel/linux-module/files/debian/control > +++ /dev/null > @@ -1,11 +0,0 @@ > -Source: @PN@ > -Section: kernel > -Priority: optional > -Standards-Version: 3.9.6 > -Build-Depends: linux-headers-@KERNEL_NAME@ > -Maintainer: ISAR project > - > -Package: @PN@ > -Architecture: any > -Depends: linux-image-@KERNEL_NAME@, kmod > -Description: @DESCRIPTION@ > diff --git a/meta/recipes-kernel/linux-module/files/debian/control.tmpl b/meta/recipes-kernel/linux-module/files/debian/control.tmpl > new file mode 100644 > index 0000000..3b3292d > --- /dev/null > +++ b/meta/recipes-kernel/linux-module/files/debian/control.tmpl > @@ -0,0 +1,11 @@ > +Source: ${PN} > +Section: kernel > +Priority: optional > +Standards-Version: 3.9.6 > +Build-Depends: linux-headers-${KERNEL_NAME} > +Maintainer: ISAR project > + > +Package: ${PN} > +Architecture: any > +Depends: linux-image-${KERNEL_NAME}, kmod > +Description: ${DESCRIPTION} > diff --git a/meta/recipes-kernel/linux-module/module.inc b/meta/recipes-kernel/linux-module/module.inc > index cb7b8ad..44edd58 100644 > --- a/meta/recipes-kernel/linux-module/module.inc > +++ b/meta/recipes-kernel/linux-module/module.inc > @@ -19,12 +19,11 @@ AUTOLOAD ?= "" > > inherit dpkg > > +TEMPLATE_FILES = "debian/control.tmpl \ > + debian/changelog.tmpl" > + > do_prepare_build() { > cp -r ${WORKDIR}/debian ${S}/ > - sed -i -e 's/@PN@/${PN}/g' -e 's/@PV@/${PV}/g' \ > - -e 's/@KERNEL_NAME@/${KERNEL_NAME}/g' \ > - -e 's/@DESCRIPTION@/${DESCRIPTION}/g' \ > - ${S}/debian/changelog ${S}/debian/control > > for module in "${AUTOLOAD}"; do > echo "echo $module >> /etc/modules" >> ${S}/debian/postinst > Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux