From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6652594697292218368 X-Received: by 2002:a1c:b4c1:: with SMTP id d184mr269559wmf.9.1549356991313; Tue, 05 Feb 2019 00:56:31 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a5d:4982:: with SMTP id r2ls3910999wrq.0.gmail; Tue, 05 Feb 2019 00:56:30 -0800 (PST) X-Google-Smtp-Source: AHgI3IbpwBUY2HsYoHxoKSgDue7a5kbJF/0+iTVv59dQimQL8dByU7pKMaCl7BJSLv2dCF5WVPx2 X-Received: by 2002:a5d:6041:: with SMTP id j1mr172535wrt.28.1549356990868; Tue, 05 Feb 2019 00:56:30 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1549356990; cv=none; d=google.com; s=arc-20160816; b=uSqGRhrJNj58yPzXB3A5zZ6j24X51yqxQaapYdWunlV1NoGQpCCWieyzE9BvYoZt6M oiFHf7L+tavZ2hkZ4kFp3IN7j4WeJd9P1MsVvoBZFT6OC23fkcWH4ynvE9Id+oMCnhwz D4S8RRInggXdpHIm+fsRhcpgi74mqBePZFZnJ2Gz7e1WjWDpOhxkVF9GzCePgDWQMaF3 1CeUi+BnbceNukAMAmEpy01uQ88cr4I7zhH9PuXP4i4KWNGdznBs1aSJQJ2VCKvLJBE9 TPLp7y3Kr/LixUoG6ca2CpBZdpP+yPDN5ypvlSeaVdCn/VDC38O6zvZPxGqI26uzhpKW pcOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:references:cc:to:subject:from; bh=lzjeF22imU3drhYA+GT7zcc+A54AHHChOaz/FdvYTAA=; b=IJie+aDbbDQjPNJYDM72diM3UQAdpXCi7S8+2+O7e1sW9PsIRnfl77A+Xl/wDy89zX 4uq3FyMM3zcGHAzdHwhEPO+un7BQcu/GyL5bDX2ZbYAI98cp1L6+/2hujEv9f0sA8Ok6 aKl8zNJVjFdDQ10wEh+GApNGC/50qcVYGM/+MDzIRR1HvTdiUaFEo7qfhPnzmXwa+3w+ tqgwsXk+6TKWJ3KaXNgV7DtR+q8AWvwNQ9a7piVRe4iQwbdfiTRhx/alPgSpbOKxw5dR vupBIxDX93mw1w10ws8CuR+1RJT64ZNlN6V5Nwm6Vp44JI3YM2D4L+5ijHlF8FeOWV4J CpUA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com Return-Path: Received: from goliath.siemens.de (goliath.siemens.de. [192.35.17.28]) by gmr-mx.google.com with ESMTPS id x188si698800wmb.3.2019.02.05.00.56.30 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 05 Feb 2019 00:56:30 -0800 (PST) Received-SPF: pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.28 as permitted sender) client-ip=192.35.17.28; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by goliath.siemens.de (8.15.2/8.15.2) with ESMTPS id x158uT7a002483 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 5 Feb 2019 09:56:29 +0100 Received: from [139.25.69.181] (linux-ses-ext02.ppmd.siemens.net [139.25.69.181]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id x158uTPF007443; Tue, 5 Feb 2019 09:56:29 +0100 From: Claudius Heine Subject: Re: [PATCH 1/1] meta: added do_transform_template task as templating system and switch To: Jan Kiszka , isar-users@googlegroups.com Cc: Claudius Heine References: <20190204135241.32000-1-claudius.heine.ext@siemens.com> <20190204135241.32000-2-claudius.heine.ext@siemens.com> Message-ID: <84453403-6092-36ed-f2ac-cc138811e108@siemens.com> Date: Tue, 5 Feb 2019 09:56:28 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: A1Q5fxlVDHsf Hi Jan, On 04/02/2019 18.55, Jan Kiszka wrote: > On 04.02.19 14:52, [ext] claudius.heine.ext@siemens.com wrote: >> From: Claudius Heine >> >> The new introduced variables are TEMPLATE_FILES and TEMPLATE_VARS. >> TEMPLATE_FILES is a list of files that should be processed in the >> `do_transform_template` task and TEMPLATE_VARS is a list of bitbake >> variable names that should be substituted in the template file. >> >> Signed-off-by: Claudius Heine >> --- >>   doc/technical_overview.md                     | 25 ++++++++ >>   meta/classes/base.bbclass                     |  1 + >>   meta/classes/dpkg-base.bbclass                |  2 +- >>   meta/classes/template.bbclass                 | 62 +++++++++++++++++++ >>   .../debian/{changelog => changelog.tmpl}      |  2 +- >>   meta/recipes-bsp/u-boot/files/debian/control  | 19 ------ >>   .../u-boot/files/debian/control.tmpl          | 19 ++++++ >>   meta/recipes-bsp/u-boot/u-boot-custom.inc     | 12 ++-- >>   .../debian/{changelog => changelog.tmpl}      |  2 +- >>   .../linux-module/files/debian/control         | 11 ---- >>   .../linux-module/files/debian/control.tmpl    | 11 ++++ >>   meta/recipes-kernel/linux-module/module.inc   |  7 +-- >>   12 files changed, 129 insertions(+), 44 deletions(-) >>   create mode 100644 meta/classes/template.bbclass >>   rename meta/recipes-bsp/u-boot/files/debian/{changelog => >> changelog.tmpl} (74%) >>   delete mode 100644 meta/recipes-bsp/u-boot/files/debian/control >>   create mode 100644 meta/recipes-bsp/u-boot/files/debian/control.tmpl >>   rename meta/recipes-kernel/linux-module/files/debian/{changelog => >> changelog.tmpl} (74%) >>   delete mode 100644 >> meta/recipes-kernel/linux-module/files/debian/control >>   create mode 100644 >> meta/recipes-kernel/linux-module/files/debian/control.tmpl >> >> diff --git a/doc/technical_overview.md b/doc/technical_overview.md >> index 7efbeee..2608727 100644 >> --- a/doc/technical_overview.md >> +++ b/doc/technical_overview.md >> @@ -283,3 +283,28 @@ cross-compilation, this is done by the same tools >> as for native compilation. >>   Depending on ISAR_CROSS_COMPILE value, additional architecture >> specifiers are >>   passed to build tools automatically, so this is absolutely >> transparent from >>   the user point of view. >> + >> +## 3.9 Additional features >> + >> +### 3.9.1 Template files >> + >> +A basic templating system implemented on top `envsubst(1)` is >> available. It >> +allows limited access to bitbake variables for file generation. >> + >> +This system is implemented in the `template.bbclass` and defines the >> +`do_transform_template` task, the `TEMPLATE_FILES` and `TEMPLATE_VARS` >> +variables. Tasks that use need to use files generated via templates need >> +to be executed after the `do_transform_template` task. >> + >> +The `TEMPLATE_FILES` variable contains a space seperated list of >> template > > separated, also below > >> +files and the `TEMPLATE_VARS` variable a space seperated list of bitbake >> +variable names. The `do_transform_template` task takes these and >> generates >> +files using `envsubst(1)`. The output files are placed in the same >> directory >> +as the template files. The name of the output files are eighter the >> name of the > > either > > (maybe turn your spell checker on in your editor ;) ) Will do :) Thx > >> +template file with `.tmpl` removed from the file name end or, if the >> template >> +file did not end with `.tmpl`, the name of the template file with >> `.out` added >> +to the end. > > I wonder if we shouldn't use '.in' instead of '.tmpl'. I don't really care about the extension. While `.in` would be more compatible with autotools, that would also cause possible template files to have an additional `.in` extension. Meaning a template for `Makefile.in` would then be called `Makefile.in.in` instead of `Makefile.in.tmpl`. I don't know, but I could make it configurable via a `TEMPLATE_EXTENSION` parameter... > >> + >> +Only template files from the `WORKDIR` are accepted. Eighter specify >> relative >> +paths based on the recipes `WORKDIR` or absolute paths containing the >> `WORKDIR` >> +in the `TEMPLATE_FILES` variable. >> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass >> index d4082de..5ab6fa4 100644 >> --- a/meta/classes/base.bbclass >> +++ b/meta/classes/base.bbclass >> @@ -25,6 +25,7 @@ die() { >>   } >>   inherit logging >> +inherit template >>   # Derived from bitbake: bitbake/classes/base.bbclass >>   addtask showdata >> diff --git a/meta/classes/dpkg-base.bbclass >> b/meta/classes/dpkg-base.bbclass >> index f1b127c..fe96850 100644 >> --- a/meta/classes/dpkg-base.bbclass >> +++ b/meta/classes/dpkg-base.bbclass >> @@ -40,7 +40,7 @@ do_prepare_build() { >>       true >>   } >> -addtask prepare_build after do_patch before do_build >> +addtask prepare_build after do_patch do_transform_template before >> do_build >>   do_prepare_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" >>   # If Isar recipes depend on each other, they typically need the package >>   # deployed to isar-apt >> diff --git a/meta/classes/template.bbclass >> b/meta/classes/template.bbclass >> new file mode 100644 >> index 0000000..716b7a5 >> --- /dev/null >> +++ b/meta/classes/template.bbclass >> @@ -0,0 +1,62 @@ >> +# This software is a part of ISAR. >> +# Copyright (C) Siemens AG, 2019 >> +# >> +# SPDX-License-Identifier: MIT >> + >> +TEMPLATE_FILES ??= "" > > Needed? You handle the empty case I think. Right that can be deleted. That had more of a documentation character, to easy see the interface to this class, but it can be deleted of course. > >> +TEMPLATE_VARS ??= "PN PV DESCRIPTION HOMEPAGE MAINTAINER KERNEL_NAME >> MACHINE \ >> +                   DISTRO_ARCH" > > I don't think MAINTAINER qualifies. DESCRIPTION and MAINTAINER have both > /some/ overlap with recipes (though both only by convention these days, > it seems), but MAINTAINER is a Debian-only thing that goes into control > files directly (usually with dummy values). Well OE uses this variable as well: https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#var-MAINTAINER We could only allow PN and PV here and add additional variables in the dpkg.bbclass so that made clear to be Debian package specific. > We should probably pull this list into out bitbake.conf, rather than > encoding it in the implementation of the class. We could, but I currently cannot think of direct advantages for that. Having those in the class, has interface documentation character. One reason for separating the template mechanism out of the base.bbclass was to have it contained together and easy to understand, extend and review. Claudius > >> + >> +do_transform_template[vardeps] = "TEMPLATE_FILES ${TEMPLATE_VARS}" >> +do_transform_template[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" >> +python do_transform_template() { >> +    import subprocess, contextlib >> + >> +    workdir = os.path.normpath(d.getVar('WORKDIR', True)) >> + >> +    template_vars = (d.getVar('TEMPLATE_VARS', True) or "").split() >> +    if len(template_vars) == 0: >> +        return >> + >> +    template_files = (d.getVar('TEMPLATE_FILES', True) or "").split() >> +    if len(template_files) == 0: >> +        return >> + >> +    cmd = "envsubst" >> +    args = " ".join("\${{{}}}".format(i) for i in template_vars) >> + >> +    # Copy current process environment and add template variables >> +    # from bitbake data store: >> +    env = os.environ.copy() >> +    for varname in template_vars: >> +        value = d.getVar(varname, True) >> +        if value: >> +            env.update({varname: value}) >> + >> +    for template_file in template_files: >> +        # Normpath and workdir checks should prevent accidential or >> +        # uninformed changes to files outside of the tmp and >> workdirectoy >> +        template_file = os.path.normpath(template_file) >> + >> +        # Convert relative paths to absolut paths based on the workdir: >> +        if not os.path.isabs(template_file): >> +            template_file = os.path.normpath(os.path.join(workdir, >> template_file)) >> + >> +        if not template_file.startswith(workdir): >> +            bb.fatal("Template file ({}) is not within workdir ({})" >> +                     .format(template_file, workdir)) >> + >> +        output_file = (os.path.splitext(template_file)[0] >> +                       if template_file.endswith(".tmpl") >> +                       else (template_file + ".out")) >> +        bb.note("{} {} [in: {} out: {}]".format(cmd, args, >> +                                                template_file, >> output_file)) >> +        with contextlib.ExitStack() as stack: >> +            input = stack.enter_context(open(template_file, 'rb')) >> +            output = stack.enter_context(open(output_file, 'wb')) >> +            process = subprocess.Popen([cmd, args], stdin=input, >> +                                       stdout=output, env=env) >> +            if process.wait() != 0: >> +                bb.fatal("processing of template failed") >> +} >> +addtask do_transform_template after do_unpack before do_build >> diff --git a/meta/recipes-bsp/u-boot/files/debian/changelog >> b/meta/recipes-bsp/u-boot/files/debian/changelog.tmpl >> similarity index 74% >> rename from meta/recipes-bsp/u-boot/files/debian/changelog >> rename to meta/recipes-bsp/u-boot/files/debian/changelog.tmpl >> index c1c3516..6e59e06 100644 >> --- a/meta/recipes-bsp/u-boot/files/debian/changelog >> +++ b/meta/recipes-bsp/u-boot/files/debian/changelog.tmpl >> @@ -1,4 +1,4 @@ >> -@PN@ (@PV@) unstable; urgency=low >> +${PN} (${PV}) unstable; urgency=low >>     * Generated package. >> diff --git a/meta/recipes-bsp/u-boot/files/debian/control >> b/meta/recipes-bsp/u-boot/files/debian/control >> deleted file mode 100644 >> index 6b4c839..0000000 >> --- a/meta/recipes-bsp/u-boot/files/debian/control >> +++ /dev/null >> @@ -1,19 +0,0 @@ >> -Source: @PN@ >> -Section: admin >> -Priority: optional >> -Standards-Version: 3.9.6 >> -Build-Depends: @BUILD_DEPENDS@ >> -Maintainer: ISAR project >> - >> -Package: u-boot-@MACHINE@ >> -Architecture: @DISTRO_ARCH@ >> -Description: @DESCRIPTION@, bootloader binaries >> - >> -Package: u-boot-@MACHINE@-dev >> -Architecture: @DISTRO_ARCH@ >> -Description: @DESCRIPTION@, bootloader libraries >> - >> -Package: u-boot-tools >> -Architecture: linux-any >> -Depends: ${shlibs:Depends}, ${misc:Depends} >> -Description: @DESCRIPTION@, companion tools >> diff --git a/meta/recipes-bsp/u-boot/files/debian/control.tmpl >> b/meta/recipes-bsp/u-boot/files/debian/control.tmpl >> new file mode 100644 >> index 0000000..5c1cc92 >> --- /dev/null >> +++ b/meta/recipes-bsp/u-boot/files/debian/control.tmpl >> @@ -0,0 +1,19 @@ >> +Source: ${PN} >> +Section: admin >> +Priority: optional >> +Standards-Version: 3.9.6 >> +Build-Depends: ${BUILD_DEPENDS} >> +Maintainer: ISAR project >> + >> +Package: u-boot-${MACHINE} >> +Architecture: ${DISTRO_ARCH} >> +Description: ${DESCRIPTION}, bootloader binaries >> + >> +Package: u-boot-${MACHINE}-dev >> +Architecture: ${DISTRO_ARCH} >> +Description: ${DESCRIPTION}, bootloader libraries >> + >> +Package: u-boot-tools >> +Architecture: linux-any >> +Depends: ${shlibs:Depends}, ${misc:Depends} >> +Description: ${DESCRIPTION}, companion tools >> diff --git a/meta/recipes-bsp/u-boot/u-boot-custom.inc >> b/meta/recipes-bsp/u-boot/u-boot-custom.inc >> index 4b38c88..c6aab43 100644 >> --- a/meta/recipes-bsp/u-boot/u-boot-custom.inc >> +++ b/meta/recipes-bsp/u-boot/u-boot-custom.inc >> @@ -15,16 +15,14 @@ inherit dpkg >>   SRC_URI += "file://debian/" >> -U_BOOT_BUILD_DEPENDS ?= "bc, bison, flex, device-tree-compiler" >> +BUILD_DEPENDS ?= "bc, bison, flex, device-tree-compiler" >> + >> +TEMPLATE_FILES = "debian/changelog.tmpl \ >> +                  debian/control.tmpl" >> +TEMPLATE_VARS += "BUILD_DEPENDS" >>   do_prepare_build() { >>       cp -r ${WORKDIR}/debian ${S}/ >> -    sed -i -e 's/@PN@/${PN}/g' -e 's/@PV@/${PV}/g' \ >> -        -e 's/@BUILD_DEPENDS@/${U_BOOT_BUILD_DEPENDS}/g' \ >> -        -e 's/@MACHINE@/${MACHINE}/g' \ >> -        -e 's/@DISTRO_ARCH@/${DISTRO_ARCH}/g' \ >> -        -e 's/@DESCRIPTION@/${DESCRIPTION}/g' \ >> -        ${S}/debian/changelog ${S}/debian/control >>       echo "${U_BOOT_BIN} /usr/lib/u-boot/${MACHINE}" > \ >>           ${S}/debian/u-boot-${MACHINE}.install >> diff --git a/meta/recipes-kernel/linux-module/files/debian/changelog >> b/meta/recipes-kernel/linux-module/files/debian/changelog.tmpl >> similarity index 74% >> rename from meta/recipes-kernel/linux-module/files/debian/changelog >> rename to meta/recipes-kernel/linux-module/files/debian/changelog.tmpl >> index c1c3516..6e59e06 100644 >> --- a/meta/recipes-kernel/linux-module/files/debian/changelog >> +++ b/meta/recipes-kernel/linux-module/files/debian/changelog.tmpl >> @@ -1,4 +1,4 @@ >> -@PN@ (@PV@) unstable; urgency=low >> +${PN} (${PV}) unstable; urgency=low >>     * Generated package. >> diff --git a/meta/recipes-kernel/linux-module/files/debian/control >> b/meta/recipes-kernel/linux-module/files/debian/control >> deleted file mode 100644 >> index 1ee634c..0000000 >> --- a/meta/recipes-kernel/linux-module/files/debian/control >> +++ /dev/null >> @@ -1,11 +0,0 @@ >> -Source: @PN@ >> -Section: kernel >> -Priority: optional >> -Standards-Version: 3.9.6 >> -Build-Depends: linux-headers-@KERNEL_NAME@ >> -Maintainer: ISAR project >> - >> -Package: @PN@ >> -Architecture: any >> -Depends: linux-image-@KERNEL_NAME@, kmod >> -Description: @DESCRIPTION@ >> diff --git >> a/meta/recipes-kernel/linux-module/files/debian/control.tmpl >> b/meta/recipes-kernel/linux-module/files/debian/control.tmpl >> new file mode 100644 >> index 0000000..3b3292d >> --- /dev/null >> +++ b/meta/recipes-kernel/linux-module/files/debian/control.tmpl >> @@ -0,0 +1,11 @@ >> +Source: ${PN} >> +Section: kernel >> +Priority: optional >> +Standards-Version: 3.9.6 >> +Build-Depends: linux-headers-${KERNEL_NAME} >> +Maintainer: ISAR project >> + >> +Package: ${PN} >> +Architecture: any >> +Depends: linux-image-${KERNEL_NAME}, kmod >> +Description: ${DESCRIPTION} >> diff --git a/meta/recipes-kernel/linux-module/module.inc >> b/meta/recipes-kernel/linux-module/module.inc >> index cb7b8ad..44edd58 100644 >> --- a/meta/recipes-kernel/linux-module/module.inc >> +++ b/meta/recipes-kernel/linux-module/module.inc >> @@ -19,12 +19,11 @@ AUTOLOAD ?= "" >>   inherit dpkg >> +TEMPLATE_FILES = "debian/control.tmpl \ >> +                  debian/changelog.tmpl" >> + >>   do_prepare_build() { >>       cp -r ${WORKDIR}/debian ${S}/ >> -    sed -i -e 's/@PN@/${PN}/g' -e 's/@PV@/${PV}/g' \ >> -           -e 's/@KERNEL_NAME@/${KERNEL_NAME}/g' \ >> -           -e 's/@DESCRIPTION@/${DESCRIPTION}/g' \ >> -        ${S}/debian/changelog ${S}/debian/control >>       for module in "${AUTOLOAD}"; do >>           echo "echo $module >> /etc/modules" >> ${S}/debian/postinst >> > > Jan > -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de