* [PATCH v3 1/1] meta: added do_transform_template task as templating system and switch
2019-02-13 15:13 [PATCH v3 0/1] Template system claudius.heine.ext
@ 2019-02-13 15:13 ` claudius.heine.ext
2019-02-14 7:54 ` [PATCH v3 0/1] Template system Maxim Yu. Osipov
1 sibling, 0 replies; 4+ messages in thread
From: claudius.heine.ext @ 2019-02-13 15:13 UTC (permalink / raw)
To: isar-users; +Cc: Claudius Heine
From: Claudius Heine <ch@denx.de>
The new introduced variables are TEMPLATE_FILES and TEMPLATE_VARS.
TEMPLATE_FILES is a list of files that should be processed in the
`do_transform_template` task and TEMPLATE_VARS is a list of bitbake
variable names that should be substituted in the template file.
Signed-off-by: Claudius Heine <ch@denx.de>
---
doc/technical_overview.md | 25 ++++++++
doc/user_manual.md | 1 +
meta/classes/base.bbclass | 1 +
meta/classes/dpkg-base.bbclass | 2 +-
meta/classes/template.bbclass | 62 +++++++++++++++++++
.../debian/{changelog => changelog.tmpl} | 2 +-
meta/recipes-bsp/u-boot/files/debian/control | 19 ------
.../u-boot/files/debian/control.tmpl | 19 ++++++
meta/recipes-bsp/u-boot/u-boot-custom.inc | 12 ++--
.../debian/{changelog => changelog.tmpl} | 2 +-
.../linux-module/files/debian/control | 11 ----
.../linux-module/files/debian/control.tmpl | 11 ++++
meta/recipes-kernel/linux-module/module.inc | 7 +--
13 files changed, 130 insertions(+), 44 deletions(-)
create mode 100644 meta/classes/template.bbclass
rename meta/recipes-bsp/u-boot/files/debian/{changelog => changelog.tmpl} (74%)
delete mode 100644 meta/recipes-bsp/u-boot/files/debian/control
create mode 100644 meta/recipes-bsp/u-boot/files/debian/control.tmpl
rename meta/recipes-kernel/linux-module/files/debian/{changelog => changelog.tmpl} (74%)
delete mode 100644 meta/recipes-kernel/linux-module/files/debian/control
create mode 100644 meta/recipes-kernel/linux-module/files/debian/control.tmpl
diff --git a/doc/technical_overview.md b/doc/technical_overview.md
index 7efbeee..0ddcabc 100644
--- a/doc/technical_overview.md
+++ b/doc/technical_overview.md
@@ -283,3 +283,28 @@ cross-compilation, this is done by the same tools as for native compilation.
Depending on ISAR_CROSS_COMPILE value, additional architecture specifiers are
passed to build tools automatically, so this is absolutely transparent from
the user point of view.
+
+## 3.9 Additional features
+
+### 3.9.1 Template files
+
+A basic templating system implemented on top `envsubst(1)` is available. It
+allows limited access to bitbake variables for file generation.
+
+This system is implemented in the `template.bbclass` and defines the
+`do_transform_template` task, the `TEMPLATE_FILES` and `TEMPLATE_VARS`
+variables. Tasks that use need to use files generated via templates need
+to be executed after the `do_transform_template` task.
+
+The `TEMPLATE_FILES` variable contains a space separated list of template
+files and the `TEMPLATE_VARS` variable a space separated list of bitbake
+variable names. The `do_transform_template` task takes these and generates
+files using `envsubst(1)`. The output files are placed in the same directory
+as the template files. The name of the output files are either the name of the
+template file with `.tmpl` removed from the file name end or, if the template
+file did not end with `.tmpl`, the name of the template file with `.out` added
+to the end.
+
+Only template files from the `WORKDIR` are accepted. Either specify relative
+paths based on the recipes `WORKDIR` or absolute paths containing the `WORKDIR`
+in the `TEMPLATE_FILES` variable.
diff --git a/doc/user_manual.md b/doc/user_manual.md
index ff4939c..c1b154f 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -68,6 +68,7 @@ qemu-user-static
binfmt-support
sudo
reprepro
+gettext-base
```
Notes:
diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 6ca93bf..4279a68 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -25,6 +25,7 @@ die() {
}
inherit logging
+inherit template
# Derived from bitbake: bitbake/classes/base.bbclass
addtask showdata
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 175dc80..320869b 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -60,7 +60,7 @@ do_prepare_build() {
true
}
-addtask prepare_build after do_patch before do_build
+addtask prepare_build after do_patch do_transform_template before do_build
do_prepare_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
# If Isar recipes depend on each other, they typically need the package
# deployed to isar-apt
diff --git a/meta/classes/template.bbclass b/meta/classes/template.bbclass
new file mode 100644
index 0000000..00d7453
--- /dev/null
+++ b/meta/classes/template.bbclass
@@ -0,0 +1,62 @@
+# This software is a part of ISAR.
+# Copyright (C) Siemens AG, 2019
+#
+# SPDX-License-Identifier: MIT
+
+TEMPLATE_FILES ??= ""
+TEMPLATE_VARS ??= "PN PV DESCRIPTION HOMEPAGE MAINTAINER KERNEL_NAME MACHINE \
+ DISTRO_ARCH"
+
+do_transform_template[vardeps] = "TEMPLATE_FILES ${TEMPLATE_VARS}"
+do_transform_template[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+python do_transform_template() {
+ import subprocess, contextlib
+
+ workdir = os.path.normpath(d.getVar('WORKDIR', True))
+
+ template_vars = (d.getVar('TEMPLATE_VARS', True) or "").split()
+ if len(template_vars) == 0:
+ return
+
+ template_files = (d.getVar('TEMPLATE_FILES', True) or "").split()
+ if len(template_files) == 0:
+ return
+
+ cmd = "envsubst"
+ args = " ".join("\${{{}}}".format(i) for i in template_vars)
+
+ # Copy current process environment and add template variables
+ # from bitbake data store:
+ env = os.environ.copy()
+ for varname in template_vars:
+ value = d.getVar(varname, True)
+ if value:
+ env.update({varname: value})
+
+ for template_file in template_files:
+ # Normpath and workdir checks should prevent accidential or
+ # uninformed changes to files outside of the tmp and workdirectoy
+ template_file = os.path.normpath(template_file)
+
+ # Convert relative paths to absolut paths based on the workdir:
+ if not os.path.isabs(template_file):
+ template_file = os.path.normpath(os.path.join(workdir, template_file))
+
+ if not template_file.startswith(workdir):
+ bb.fatal("Template file ({}) is not within workdir ({})"
+ .format(template_file, workdir))
+
+ output_file = (os.path.splitext(template_file)[0]
+ if template_file.endswith(".tmpl")
+ else (template_file + ".out"))
+ bb.note("{} {} [in: {} out: {}]".format(cmd, args,
+ template_file, output_file))
+ with contextlib.ExitStack() as stack:
+ input = stack.enter_context(open(template_file, 'rb'))
+ output = stack.enter_context(open(output_file, 'wb'))
+ process = stack.enter_context(subprocess.Popen([cmd, args], stdin=input,
+ stdout=output, env=env))
+ if process.wait() != 0:
+ bb.fatal("processing of template failed")
+}
+addtask do_transform_template after do_unpack
diff --git a/meta/recipes-bsp/u-boot/files/debian/changelog b/meta/recipes-bsp/u-boot/files/debian/changelog.tmpl
similarity index 74%
rename from meta/recipes-bsp/u-boot/files/debian/changelog
rename to meta/recipes-bsp/u-boot/files/debian/changelog.tmpl
index c1c3516..6e59e06 100644
--- a/meta/recipes-bsp/u-boot/files/debian/changelog
+++ b/meta/recipes-bsp/u-boot/files/debian/changelog.tmpl
@@ -1,4 +1,4 @@
-@PN@ (@PV@) unstable; urgency=low
+${PN} (${PV}) unstable; urgency=low
* Generated package.
diff --git a/meta/recipes-bsp/u-boot/files/debian/control b/meta/recipes-bsp/u-boot/files/debian/control
deleted file mode 100644
index 6b4c839..0000000
--- a/meta/recipes-bsp/u-boot/files/debian/control
+++ /dev/null
@@ -1,19 +0,0 @@
-Source: @PN@
-Section: admin
-Priority: optional
-Standards-Version: 3.9.6
-Build-Depends: @BUILD_DEPENDS@
-Maintainer: ISAR project <isar-users@googlegroups.com>
-
-Package: u-boot-@MACHINE@
-Architecture: @DISTRO_ARCH@
-Description: @DESCRIPTION@, bootloader binaries
-
-Package: u-boot-@MACHINE@-dev
-Architecture: @DISTRO_ARCH@
-Description: @DESCRIPTION@, bootloader libraries
-
-Package: u-boot-tools
-Architecture: linux-any
-Depends: ${shlibs:Depends}, ${misc:Depends}
-Description: @DESCRIPTION@, companion tools
diff --git a/meta/recipes-bsp/u-boot/files/debian/control.tmpl b/meta/recipes-bsp/u-boot/files/debian/control.tmpl
new file mode 100644
index 0000000..5c1cc92
--- /dev/null
+++ b/meta/recipes-bsp/u-boot/files/debian/control.tmpl
@@ -0,0 +1,19 @@
+Source: ${PN}
+Section: admin
+Priority: optional
+Standards-Version: 3.9.6
+Build-Depends: ${BUILD_DEPENDS}
+Maintainer: ISAR project <isar-users@googlegroups.com>
+
+Package: u-boot-${MACHINE}
+Architecture: ${DISTRO_ARCH}
+Description: ${DESCRIPTION}, bootloader binaries
+
+Package: u-boot-${MACHINE}-dev
+Architecture: ${DISTRO_ARCH}
+Description: ${DESCRIPTION}, bootloader libraries
+
+Package: u-boot-tools
+Architecture: linux-any
+Depends: ${shlibs:Depends}, ${misc:Depends}
+Description: ${DESCRIPTION}, companion tools
diff --git a/meta/recipes-bsp/u-boot/u-boot-custom.inc b/meta/recipes-bsp/u-boot/u-boot-custom.inc
index 4b38c88..c6aab43 100644
--- a/meta/recipes-bsp/u-boot/u-boot-custom.inc
+++ b/meta/recipes-bsp/u-boot/u-boot-custom.inc
@@ -15,16 +15,14 @@ inherit dpkg
SRC_URI += "file://debian/"
-U_BOOT_BUILD_DEPENDS ?= "bc, bison, flex, device-tree-compiler"
+BUILD_DEPENDS ?= "bc, bison, flex, device-tree-compiler"
+
+TEMPLATE_FILES = "debian/changelog.tmpl \
+ debian/control.tmpl"
+TEMPLATE_VARS += "BUILD_DEPENDS"
do_prepare_build() {
cp -r ${WORKDIR}/debian ${S}/
- sed -i -e 's/@PN@/${PN}/g' -e 's/@PV@/${PV}/g' \
- -e 's/@BUILD_DEPENDS@/${U_BOOT_BUILD_DEPENDS}/g' \
- -e 's/@MACHINE@/${MACHINE}/g' \
- -e 's/@DISTRO_ARCH@/${DISTRO_ARCH}/g' \
- -e 's/@DESCRIPTION@/${DESCRIPTION}/g' \
- ${S}/debian/changelog ${S}/debian/control
echo "${U_BOOT_BIN} /usr/lib/u-boot/${MACHINE}" > \
${S}/debian/u-boot-${MACHINE}.install
diff --git a/meta/recipes-kernel/linux-module/files/debian/changelog b/meta/recipes-kernel/linux-module/files/debian/changelog.tmpl
similarity index 74%
rename from meta/recipes-kernel/linux-module/files/debian/changelog
rename to meta/recipes-kernel/linux-module/files/debian/changelog.tmpl
index c1c3516..6e59e06 100644
--- a/meta/recipes-kernel/linux-module/files/debian/changelog
+++ b/meta/recipes-kernel/linux-module/files/debian/changelog.tmpl
@@ -1,4 +1,4 @@
-@PN@ (@PV@) unstable; urgency=low
+${PN} (${PV}) unstable; urgency=low
* Generated package.
diff --git a/meta/recipes-kernel/linux-module/files/debian/control b/meta/recipes-kernel/linux-module/files/debian/control
deleted file mode 100644
index 1ee634c..0000000
--- a/meta/recipes-kernel/linux-module/files/debian/control
+++ /dev/null
@@ -1,11 +0,0 @@
-Source: @PN@
-Section: kernel
-Priority: optional
-Standards-Version: 3.9.6
-Build-Depends: linux-headers-@KERNEL_NAME@
-Maintainer: ISAR project <isar-users@googlegroups.com>
-
-Package: @PN@
-Architecture: any
-Depends: linux-image-@KERNEL_NAME@, kmod
-Description: @DESCRIPTION@
diff --git a/meta/recipes-kernel/linux-module/files/debian/control.tmpl b/meta/recipes-kernel/linux-module/files/debian/control.tmpl
new file mode 100644
index 0000000..3b3292d
--- /dev/null
+++ b/meta/recipes-kernel/linux-module/files/debian/control.tmpl
@@ -0,0 +1,11 @@
+Source: ${PN}
+Section: kernel
+Priority: optional
+Standards-Version: 3.9.6
+Build-Depends: linux-headers-${KERNEL_NAME}
+Maintainer: ISAR project <isar-users@googlegroups.com>
+
+Package: ${PN}
+Architecture: any
+Depends: linux-image-${KERNEL_NAME}, kmod
+Description: ${DESCRIPTION}
diff --git a/meta/recipes-kernel/linux-module/module.inc b/meta/recipes-kernel/linux-module/module.inc
index cb7b8ad..44edd58 100644
--- a/meta/recipes-kernel/linux-module/module.inc
+++ b/meta/recipes-kernel/linux-module/module.inc
@@ -19,12 +19,11 @@ AUTOLOAD ?= ""
inherit dpkg
+TEMPLATE_FILES = "debian/control.tmpl \
+ debian/changelog.tmpl"
+
do_prepare_build() {
cp -r ${WORKDIR}/debian ${S}/
- sed -i -e 's/@PN@/${PN}/g' -e 's/@PV@/${PV}/g' \
- -e 's/@KERNEL_NAME@/${KERNEL_NAME}/g' \
- -e 's/@DESCRIPTION@/${DESCRIPTION}/g' \
- ${S}/debian/changelog ${S}/debian/control
for module in "${AUTOLOAD}"; do
echo "echo $module >> /etc/modules" >> ${S}/debian/postinst
--
2.20.1
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [PATCH v3 0/1] Template system
2019-02-13 15:13 [PATCH v3 0/1] Template system claudius.heine.ext
2019-02-13 15:13 ` [PATCH v3 1/1] meta: added do_transform_template task as templating system and switch claudius.heine.ext
@ 2019-02-14 7:54 ` Maxim Yu. Osipov
2019-02-14 8:07 ` Claudius Heine
1 sibling, 1 reply; 4+ messages in thread
From: Maxim Yu. Osipov @ 2019-02-14 7:54 UTC (permalink / raw)
To: claudius.heine.ext, isar-users; +Cc: Claudius Heine
Hi Claudius,
Good news:
I've installed gettext-base in ilbers CI and with this version of patch
your series applied over current next was built OK:
http://isar-build.org:8080/job/isar_mosipov_next/142/consoleFull
Only smoke test for qemui386-buster failed - but this is not relevant to
your patches (this sporadically happens and needs to be investigated
separately).
Bad news:
So far build of de0-nano-soc-stretch target with enabled
CROSS_COMPILATION was not included into CI.
After upgrade of CI server I'll include de0-nano-soc-stretch target into
CI "fast" build as this is a supported target which in particular builds
linux kernel and u-boot.
The corresponding patch will be sent afterwards.
So I tried to build de0-nano-soc-stretch target (on my branch
https://github.com/ilbers/isar/commits/mosipov-next) with enabled
CROSS_COMPILATION - and it fails after applying your patches. See log
below. Build of de0-nano-soc-stretch with enabled CROSS_COMPILATION on
current 'next' passes OK.
========
bitbake multiconfig:de0-nano-soc-stretch:isar-image-base
<snip>
NOTE: Executing RunQueue Tasks
ERROR: mc:de0-nano-soc-stretch:u-boot-2018.09-r0 do_install_builddeps:
Function failed: do_install_builddeps (log file is located at
/home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-armhf/u-boot-2018.09-r0/temp/log.do_install_builddeps.5395)
ERROR: Logfile of failure stored in:
/home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-armhf/u-boot-2018.09-r0/temp/log.do_install_builddeps.5395
Log data follows:
| DEBUG: Executing shell function do_install_builddeps
| Get:1 file:/isar-apt isar InRelease
| Ign:1 file:/isar-apt isar InRelease
| Get:2 file:/isar-apt isar Release [3563 B]
| Get:2 file:/isar-apt isar Release [3563 B]
| Get:3 file:/isar-apt isar Release.gpg
| Ign:3 file:/isar-apt isar Release.gpg
| Get:4 file:/isar-apt isar/main armhf Packages [544 B]
| Reading package lists...
| mk-build-deps: warning: debian/changelog(l1): badly formatted
heading line
| LINE: ${PN} (${PV}) unstable; urgency=low
| mk-build-deps: warning: debian/changelog(l2): found blank line
where expected first heading
| mk-build-deps: warning: debian/changelog(l3): found change data
where expected first heading
| LINE: * Generated package.
| mk-build-deps: warning: unknown information field '' in input data in
parsed version of changelog
| hostname: Name or service not known
| dpkg-architecture: warning: specified GNU system type
arm-linux-gnueabihf does not match CC system type x86_64-linux-gnu, try
setting a correct CC environment variable
| dh_testdir
| dh_testroot
| dh_prep
| dh_testdir
| dh_testroot
| dh_install
| dh_install: Compatibility levels before 9 are deprecated (level 7 in use)
| dh_installdocs
| dh_installdocs: Compatibility levels before 9 are deprecated (level 7
in use)
| dh_installchangelogs
| dpkg-parsechangelog: warning: debian/changelog(l1): badly
formatted heading line
| LINE: ${PN}-cross-build-deps (1.0) unstable; urgency=low
| dpkg-parsechangelog: warning: debian/changelog(l2): found blank
line where expected first heading
| dpkg-parsechangelog: warning: debian/changelog(l3): found change
data where expected first heading
| LINE: * First version
| dpkg-parsechangelog: warning: unknown information field '' in input
data in parsed version of changelog
| dh_compress
| find: 'debian/-cross-build-deps': No such file or directory
| dh_fixperms
| dh_installdeb
| dh_installdeb: Compatibility levels before 9 are deprecated (level 7
in use)
| dh_gencontrol
| dpkg-gencontrol: error: illegal package name '${PN}-cross-build-deps':
character '$' not allowed
| dh_gencontrol: dpkg-gencontrol -p${PN}-cross-build-deps
-ldebian/changelog -Tdebian/${PN}-cross-build-deps.substvars
-Pdebian/${PN}-cross-build-deps returned exit code 255
| debian/rules:20: recipe for target 'binary-indep' failed
| make: *** [binary-indep] Error 2
| Error in the build process: exit status 2
| dpkg: error: cannot access archive
'${PN}-cross-build-deps_1.0_armhf.deb': No such file or directory
| mk-build-deps: dpkg --unpack failed
| WARNING: exit code 2 from a shell command.
| ERROR: Function failed: do_install_builddeps (log file is located at
/home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-armhf/u-boot-2018.09-r0/temp/log.do_install_builddeps.5395)
======
On 2/13/19 4:13 PM, claudius.heine.ext@siemens.com wrote:
> From: Claudius Heine <ch@denx.de>
>
> Hi,
>
> so after having access to the ilbers ci I could find some issues with
> this patch.
>
> I will try to describe the issue I think happened here:
>
> Initially I added the 'do_transform_template' task with a 'before
> do_build' that normally just triggers the template generation process if
> the default task 'do_build' is run. The 'do_transform_template' task has
> the default '${DISTRO}-${DISTRO_ARCH}' stamp-extra-info that we have to
> add to pretty much all new tasks in isar.
>
> That might be ok for normal recipes, but the buildchroot-* recipes add
> stuff to the virtual default `do_build` and that now depend on the
> `do_transform_template` task and its dependencies.
>
> Currently that is a bit to complex for me to exactly figure out how
> those errors get triggered, but since that works in the internal CI, but
> not on ilbers, that messes up the bitbake task order.
>
> So my solution is to remove the 'before do_build', now
> 'do_transform_template' is no longer executed per default. 'dpkg-base'
> and the image classes add their own dependencies if they need that task.
>
> Currently the build a ilbers ci still fails, because apparently
> gettext-base is not installed there. I added gettext-base as a
> dependency to the user_manual.md
>
> regards,
> Claudius
>
> Changes from v2:
> - put subprocess into context manager
> - added 'gettext-base' isar dependency
> - remove 'before do_build' from addtask
>
> Changes from v1:
> - fixed spelling mistakes of documentation
>
> Claudius Heine (1):
> meta: added do_transform_template task as templating system and switch
>
> doc/technical_overview.md | 25 ++++++++
> doc/user_manual.md | 1 +
> meta/classes/base.bbclass | 1 +
> meta/classes/dpkg-base.bbclass | 2 +-
> meta/classes/template.bbclass | 62 +++++++++++++++++++
> .../debian/{changelog => changelog.tmpl} | 2 +-
> meta/recipes-bsp/u-boot/files/debian/control | 19 ------
> .../u-boot/files/debian/control.tmpl | 19 ++++++
> meta/recipes-bsp/u-boot/u-boot-custom.inc | 12 ++--
> .../debian/{changelog => changelog.tmpl} | 2 +-
> .../linux-module/files/debian/control | 11 ----
> .../linux-module/files/debian/control.tmpl | 11 ++++
> meta/recipes-kernel/linux-module/module.inc | 7 +--
> 13 files changed, 130 insertions(+), 44 deletions(-)
> create mode 100644 meta/classes/template.bbclass
> rename meta/recipes-bsp/u-boot/files/debian/{changelog => changelog.tmpl} (74%)
> delete mode 100644 meta/recipes-bsp/u-boot/files/debian/control
> create mode 100644 meta/recipes-bsp/u-boot/files/debian/control.tmpl
> rename meta/recipes-kernel/linux-module/files/debian/{changelog => changelog.tmpl} (74%)
> delete mode 100644 meta/recipes-kernel/linux-module/files/debian/control
> create mode 100644 meta/recipes-kernel/linux-module/files/debian/control.tmpl
>
--
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 4+ messages in thread