* [PATCH v4 0/1] Template system
@ 2019-02-14 11:28 claudius.heine.ext
2019-02-14 11:28 ` [PATCH v4 1/1] meta: added do_transform_template task as templating system and switch claudius.heine.ext
2019-02-15 14:45 ` [PATCH v4 0/1] Template system Maxim Yu. Osipov
0 siblings, 2 replies; 9+ messages in thread
From: claudius.heine.ext @ 2019-02-14 11:28 UTC (permalink / raw)
To: isar-users; +Cc: Claudius Heine
From: Claudius Heine <ch@denx.de>
Hi,
apparently the '??=' in the template class prevented the '+=' in the
u-boot class to properly extend the TEMPLATE_VARS variable. I now
changed that to '?=' and that solved this issue.
regards,
Claudius
Changes from v3:
- replaced '??=' with '?=' in template.bbclass
Changes from v2:
- put subprocess into context manager
- added 'gettext-base' isar dependency
- remove 'before do_build' from addtask
Changes from v1:
- fixed spelling mistakes of documentation
Claudius Heine (1):
meta: added do_transform_template task as templating system and switch
doc/technical_overview.md | 25 ++++++++
doc/user_manual.md | 1 +
meta/classes/base.bbclass | 1 +
meta/classes/dpkg-base.bbclass | 2 +-
meta/classes/template.bbclass | 62 +++++++++++++++++++
.../debian/{changelog => changelog.tmpl} | 2 +-
meta/recipes-bsp/u-boot/files/debian/control | 19 ------
.../u-boot/files/debian/control.tmpl | 19 ++++++
meta/recipes-bsp/u-boot/u-boot-custom.inc | 12 ++--
.../debian/{changelog => changelog.tmpl} | 2 +-
.../linux-module/files/debian/control | 11 ----
.../linux-module/files/debian/control.tmpl | 11 ++++
meta/recipes-kernel/linux-module/module.inc | 7 +--
13 files changed, 130 insertions(+), 44 deletions(-)
create mode 100644 meta/classes/template.bbclass
rename meta/recipes-bsp/u-boot/files/debian/{changelog => changelog.tmpl} (74%)
delete mode 100644 meta/recipes-bsp/u-boot/files/debian/control
create mode 100644 meta/recipes-bsp/u-boot/files/debian/control.tmpl
rename meta/recipes-kernel/linux-module/files/debian/{changelog => changelog.tmpl} (74%)
delete mode 100644 meta/recipes-kernel/linux-module/files/debian/control
create mode 100644 meta/recipes-kernel/linux-module/files/debian/control.tmpl
--
2.20.1
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH v4 1/1] meta: added do_transform_template task as templating system and switch 2019-02-14 11:28 [PATCH v4 0/1] Template system claudius.heine.ext @ 2019-02-14 11:28 ` claudius.heine.ext 2019-03-24 18:28 ` Jan Kiszka 2019-02-15 14:45 ` [PATCH v4 0/1] Template system Maxim Yu. Osipov 1 sibling, 1 reply; 9+ messages in thread From: claudius.heine.ext @ 2019-02-14 11:28 UTC (permalink / raw) To: isar-users; +Cc: Claudius Heine From: Claudius Heine <ch@denx.de> The new introduced variables are TEMPLATE_FILES and TEMPLATE_VARS. TEMPLATE_FILES is a list of files that should be processed in the `do_transform_template` task and TEMPLATE_VARS is a list of bitbake variable names that should be substituted in the template file. Signed-off-by: Claudius Heine <ch@denx.de> --- doc/technical_overview.md | 25 ++++++++ doc/user_manual.md | 1 + meta/classes/base.bbclass | 1 + meta/classes/dpkg-base.bbclass | 2 +- meta/classes/template.bbclass | 62 +++++++++++++++++++ .../debian/{changelog => changelog.tmpl} | 2 +- meta/recipes-bsp/u-boot/files/debian/control | 19 ------ .../u-boot/files/debian/control.tmpl | 19 ++++++ meta/recipes-bsp/u-boot/u-boot-custom.inc | 12 ++-- .../debian/{changelog => changelog.tmpl} | 2 +- .../linux-module/files/debian/control | 11 ---- .../linux-module/files/debian/control.tmpl | 11 ++++ meta/recipes-kernel/linux-module/module.inc | 7 +-- 13 files changed, 130 insertions(+), 44 deletions(-) create mode 100644 meta/classes/template.bbclass rename meta/recipes-bsp/u-boot/files/debian/{changelog => changelog.tmpl} (74%) delete mode 100644 meta/recipes-bsp/u-boot/files/debian/control create mode 100644 meta/recipes-bsp/u-boot/files/debian/control.tmpl rename meta/recipes-kernel/linux-module/files/debian/{changelog => changelog.tmpl} (74%) delete mode 100644 meta/recipes-kernel/linux-module/files/debian/control create mode 100644 meta/recipes-kernel/linux-module/files/debian/control.tmpl diff --git a/doc/technical_overview.md b/doc/technical_overview.md index 7efbeee..0ddcabc 100644 --- a/doc/technical_overview.md +++ b/doc/technical_overview.md @@ -283,3 +283,28 @@ cross-compilation, this is done by the same tools as for native compilation. Depending on ISAR_CROSS_COMPILE value, additional architecture specifiers are passed to build tools automatically, so this is absolutely transparent from the user point of view. + +## 3.9 Additional features + +### 3.9.1 Template files + +A basic templating system implemented on top `envsubst(1)` is available. It +allows limited access to bitbake variables for file generation. + +This system is implemented in the `template.bbclass` and defines the +`do_transform_template` task, the `TEMPLATE_FILES` and `TEMPLATE_VARS` +variables. Tasks that use need to use files generated via templates need +to be executed after the `do_transform_template` task. + +The `TEMPLATE_FILES` variable contains a space separated list of template +files and the `TEMPLATE_VARS` variable a space separated list of bitbake +variable names. The `do_transform_template` task takes these and generates +files using `envsubst(1)`. The output files are placed in the same directory +as the template files. The name of the output files are either the name of the +template file with `.tmpl` removed from the file name end or, if the template +file did not end with `.tmpl`, the name of the template file with `.out` added +to the end. + +Only template files from the `WORKDIR` are accepted. Either specify relative +paths based on the recipes `WORKDIR` or absolute paths containing the `WORKDIR` +in the `TEMPLATE_FILES` variable. diff --git a/doc/user_manual.md b/doc/user_manual.md index ff4939c..c1b154f 100644 --- a/doc/user_manual.md +++ b/doc/user_manual.md @@ -68,6 +68,7 @@ qemu-user-static binfmt-support sudo reprepro +gettext-base ``` Notes: diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass index 6ca93bf..4279a68 100644 --- a/meta/classes/base.bbclass +++ b/meta/classes/base.bbclass @@ -25,6 +25,7 @@ die() { } inherit logging +inherit template # Derived from bitbake: bitbake/classes/base.bbclass addtask showdata diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index 175dc80..320869b 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -60,7 +60,7 @@ do_prepare_build() { true } -addtask prepare_build after do_patch before do_build +addtask prepare_build after do_patch do_transform_template before do_build do_prepare_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" # If Isar recipes depend on each other, they typically need the package # deployed to isar-apt diff --git a/meta/classes/template.bbclass b/meta/classes/template.bbclass new file mode 100644 index 0000000..324511a --- /dev/null +++ b/meta/classes/template.bbclass @@ -0,0 +1,62 @@ +# This software is a part of ISAR. +# Copyright (C) Siemens AG, 2019 +# +# SPDX-License-Identifier: MIT + +TEMPLATE_FILES ?= "" +TEMPLATE_VARS ?= "PN PV DESCRIPTION HOMEPAGE MAINTAINER KERNEL_NAME MACHINE \ + DISTRO_ARCH" + +do_transform_template[vardeps] = "TEMPLATE_FILES ${TEMPLATE_VARS}" +do_transform_template[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" +python do_transform_template() { + import subprocess, contextlib + + workdir = os.path.normpath(d.getVar('WORKDIR', True)) + + template_vars = (d.getVar('TEMPLATE_VARS', True) or "").split() + if len(template_vars) == 0: + return + + template_files = (d.getVar('TEMPLATE_FILES', True) or "").split() + if len(template_files) == 0: + return + + cmd = "envsubst" + args = " ".join("\${{{}}}".format(i) for i in template_vars) + + # Copy current process environment and add template variables + # from bitbake data store: + env = os.environ.copy() + for varname in template_vars: + value = d.getVar(varname, True) + if value: + env.update({varname: value}) + + for template_file in template_files: + # Normpath and workdir checks should prevent accidential or + # uninformed changes to files outside of the tmp and workdirectoy + template_file = os.path.normpath(template_file) + + # Convert relative paths to absolut paths based on the workdir: + if not os.path.isabs(template_file): + template_file = os.path.normpath(os.path.join(workdir, template_file)) + + if not template_file.startswith(workdir): + bb.fatal("Template file ({}) is not within workdir ({})" + .format(template_file, workdir)) + + output_file = (os.path.splitext(template_file)[0] + if template_file.endswith(".tmpl") + else (template_file + ".out")) + bb.note("{} {} [in: {} out: {}]".format(cmd, args, + template_file, output_file)) + with contextlib.ExitStack() as stack: + input = stack.enter_context(open(template_file, 'rb')) + output = stack.enter_context(open(output_file, 'wb')) + process = stack.enter_context(subprocess.Popen([cmd, args], stdin=input, + stdout=output, env=env)) + if process.wait() != 0: + bb.fatal("processing of template failed") +} +addtask do_transform_template after do_unpack diff --git a/meta/recipes-bsp/u-boot/files/debian/changelog b/meta/recipes-bsp/u-boot/files/debian/changelog.tmpl similarity index 74% rename from meta/recipes-bsp/u-boot/files/debian/changelog rename to meta/recipes-bsp/u-boot/files/debian/changelog.tmpl index c1c3516..6e59e06 100644 --- a/meta/recipes-bsp/u-boot/files/debian/changelog +++ b/meta/recipes-bsp/u-boot/files/debian/changelog.tmpl @@ -1,4 +1,4 @@ -@PN@ (@PV@) unstable; urgency=low +${PN} (${PV}) unstable; urgency=low * Generated package. diff --git a/meta/recipes-bsp/u-boot/files/debian/control b/meta/recipes-bsp/u-boot/files/debian/control deleted file mode 100644 index 6b4c839..0000000 --- a/meta/recipes-bsp/u-boot/files/debian/control +++ /dev/null @@ -1,19 +0,0 @@ -Source: @PN@ -Section: admin -Priority: optional -Standards-Version: 3.9.6 -Build-Depends: @BUILD_DEPENDS@ -Maintainer: ISAR project <isar-users@googlegroups.com> - -Package: u-boot-@MACHINE@ -Architecture: @DISTRO_ARCH@ -Description: @DESCRIPTION@, bootloader binaries - -Package: u-boot-@MACHINE@-dev -Architecture: @DISTRO_ARCH@ -Description: @DESCRIPTION@, bootloader libraries - -Package: u-boot-tools -Architecture: linux-any -Depends: ${shlibs:Depends}, ${misc:Depends} -Description: @DESCRIPTION@, companion tools diff --git a/meta/recipes-bsp/u-boot/files/debian/control.tmpl b/meta/recipes-bsp/u-boot/files/debian/control.tmpl new file mode 100644 index 0000000..5c1cc92 --- /dev/null +++ b/meta/recipes-bsp/u-boot/files/debian/control.tmpl @@ -0,0 +1,19 @@ +Source: ${PN} +Section: admin +Priority: optional +Standards-Version: 3.9.6 +Build-Depends: ${BUILD_DEPENDS} +Maintainer: ISAR project <isar-users@googlegroups.com> + +Package: u-boot-${MACHINE} +Architecture: ${DISTRO_ARCH} +Description: ${DESCRIPTION}, bootloader binaries + +Package: u-boot-${MACHINE}-dev +Architecture: ${DISTRO_ARCH} +Description: ${DESCRIPTION}, bootloader libraries + +Package: u-boot-tools +Architecture: linux-any +Depends: ${shlibs:Depends}, ${misc:Depends} +Description: ${DESCRIPTION}, companion tools diff --git a/meta/recipes-bsp/u-boot/u-boot-custom.inc b/meta/recipes-bsp/u-boot/u-boot-custom.inc index 4b38c88..c6aab43 100644 --- a/meta/recipes-bsp/u-boot/u-boot-custom.inc +++ b/meta/recipes-bsp/u-boot/u-boot-custom.inc @@ -15,16 +15,14 @@ inherit dpkg SRC_URI += "file://debian/" -U_BOOT_BUILD_DEPENDS ?= "bc, bison, flex, device-tree-compiler" +BUILD_DEPENDS ?= "bc, bison, flex, device-tree-compiler" + +TEMPLATE_FILES = "debian/changelog.tmpl \ + debian/control.tmpl" +TEMPLATE_VARS += "BUILD_DEPENDS" do_prepare_build() { cp -r ${WORKDIR}/debian ${S}/ - sed -i -e 's/@PN@/${PN}/g' -e 's/@PV@/${PV}/g' \ - -e 's/@BUILD_DEPENDS@/${U_BOOT_BUILD_DEPENDS}/g' \ - -e 's/@MACHINE@/${MACHINE}/g' \ - -e 's/@DISTRO_ARCH@/${DISTRO_ARCH}/g' \ - -e 's/@DESCRIPTION@/${DESCRIPTION}/g' \ - ${S}/debian/changelog ${S}/debian/control echo "${U_BOOT_BIN} /usr/lib/u-boot/${MACHINE}" > \ ${S}/debian/u-boot-${MACHINE}.install diff --git a/meta/recipes-kernel/linux-module/files/debian/changelog b/meta/recipes-kernel/linux-module/files/debian/changelog.tmpl similarity index 74% rename from meta/recipes-kernel/linux-module/files/debian/changelog rename to meta/recipes-kernel/linux-module/files/debian/changelog.tmpl index c1c3516..6e59e06 100644 --- a/meta/recipes-kernel/linux-module/files/debian/changelog +++ b/meta/recipes-kernel/linux-module/files/debian/changelog.tmpl @@ -1,4 +1,4 @@ -@PN@ (@PV@) unstable; urgency=low +${PN} (${PV}) unstable; urgency=low * Generated package. diff --git a/meta/recipes-kernel/linux-module/files/debian/control b/meta/recipes-kernel/linux-module/files/debian/control deleted file mode 100644 index 1ee634c..0000000 --- a/meta/recipes-kernel/linux-module/files/debian/control +++ /dev/null @@ -1,11 +0,0 @@ -Source: @PN@ -Section: kernel -Priority: optional -Standards-Version: 3.9.6 -Build-Depends: linux-headers-@KERNEL_NAME@ -Maintainer: ISAR project <isar-users@googlegroups.com> - -Package: @PN@ -Architecture: any -Depends: linux-image-@KERNEL_NAME@, kmod -Description: @DESCRIPTION@ diff --git a/meta/recipes-kernel/linux-module/files/debian/control.tmpl b/meta/recipes-kernel/linux-module/files/debian/control.tmpl new file mode 100644 index 0000000..3b3292d --- /dev/null +++ b/meta/recipes-kernel/linux-module/files/debian/control.tmpl @@ -0,0 +1,11 @@ +Source: ${PN} +Section: kernel +Priority: optional +Standards-Version: 3.9.6 +Build-Depends: linux-headers-${KERNEL_NAME} +Maintainer: ISAR project <isar-users@googlegroups.com> + +Package: ${PN} +Architecture: any +Depends: linux-image-${KERNEL_NAME}, kmod +Description: ${DESCRIPTION} diff --git a/meta/recipes-kernel/linux-module/module.inc b/meta/recipes-kernel/linux-module/module.inc index cb7b8ad..44edd58 100644 --- a/meta/recipes-kernel/linux-module/module.inc +++ b/meta/recipes-kernel/linux-module/module.inc @@ -19,12 +19,11 @@ AUTOLOAD ?= "" inherit dpkg +TEMPLATE_FILES = "debian/control.tmpl \ + debian/changelog.tmpl" + do_prepare_build() { cp -r ${WORKDIR}/debian ${S}/ - sed -i -e 's/@PN@/${PN}/g' -e 's/@PV@/${PV}/g' \ - -e 's/@KERNEL_NAME@/${KERNEL_NAME}/g' \ - -e 's/@DESCRIPTION@/${DESCRIPTION}/g' \ - ${S}/debian/changelog ${S}/debian/control for module in "${AUTOLOAD}"; do echo "echo $module >> /etc/modules" >> ${S}/debian/postinst -- 2.20.1 ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v4 1/1] meta: added do_transform_template task as templating system and switch 2019-02-14 11:28 ` [PATCH v4 1/1] meta: added do_transform_template task as templating system and switch claudius.heine.ext @ 2019-03-24 18:28 ` Jan Kiszka 2019-03-25 7:53 ` Claudius Heine 0 siblings, 1 reply; 9+ messages in thread From: Jan Kiszka @ 2019-03-24 18:28 UTC (permalink / raw) To: [ext] claudius.heine.ext@siemens.com, isar-users; +Cc: Claudius Heine On 14.02.19 12:28, [ext] claudius.heine.ext@siemens.com wrote: > From: Claudius Heine <ch@denx.de> > > The new introduced variables are TEMPLATE_FILES and TEMPLATE_VARS. > TEMPLATE_FILES is a list of files that should be processed in the > `do_transform_template` task and TEMPLATE_VARS is a list of bitbake > variable names that should be substituted in the template file. > Bad news: This patch introduced a regression into the release (sorry, didn't manage to test earlier). It destroys the reuse of self-built identical packages for different targets. Namely, jailhouse-images now tries to build the shared (per arch) kernel multiple times, and that fails miserably because there is a common folder for all of them. The issue disappears when removing the do_transform_template task from the chain. Digging deeper, the reason is that do_transform_template is not shared but run multiple times (even if nothing has to be done), forcing all succeeding tasks into private mode as well. I suspect that's because bitbake cannot assess if the output of the task python function is identical. I guess we need to help bitbake here. Any idea how? Own stamping logic? Jan ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v4 1/1] meta: added do_transform_template task as templating system and switch 2019-03-24 18:28 ` Jan Kiszka @ 2019-03-25 7:53 ` Claudius Heine 2019-03-25 8:13 ` Jan Kiszka 0 siblings, 1 reply; 9+ messages in thread From: Claudius Heine @ 2019-03-25 7:53 UTC (permalink / raw) To: Jan Kiszka, isar-users; +Cc: Claudius Heine Hi Jan, On Sun, 2019-03-24 at 19:28 +0100, Jan Kiszka wrote: > On 14.02.19 12:28, [ext] claudius.heine.ext@siemens.com wrote: > > From: Claudius Heine <ch@denx.de> > > > > The new introduced variables are TEMPLATE_FILES and TEMPLATE_VARS. > > TEMPLATE_FILES is a list of files that should be processed in the > > `do_transform_template` task and TEMPLATE_VARS is a list of bitbake > > variable names that should be substituted in the template file. > > > > Bad news: This patch introduced a regression into the release (sorry, > didn't > manage to test earlier). It destroys the reuse of self-built > identical packages > for different targets. Namely, jailhouse-images now tries to build > the shared > (per arch) kernel multiple times, and that fails miserably because > there is a > common folder for all of them. The issue disappears when removing the > do_transform_template task from the chain. Hmm, have you looked at the resulting 'TEMPLATE_VARS' variable? All variables in there need to be the same in order for it to have the same stamp. If that is not the case then maybe overwriting it instead of appending it might work. These are the defaults set in 'template.bbclass': TEMPLATE_VARS ?= "PN PV DESCRIPTION HOMEPAGE MAINTAINER KERNEL_NAME MACHINE DISTRO_ARCH" If I understand you correctly then your recipe should not be dependant on the MACHINE? So either remove MACHINE 'TEMPLATE_VARS_remove = "MACHINE"' or just set a 'TEMPLATE_VARS' that you really need in your template files. Cheers, Claudius > > Digging deeper, the reason is that do_transform_template is not > shared but run > multiple times (even if nothing has to be done), forcing all > succeeding tasks > into private mode as well. I suspect that's because bitbake cannot > assess if the > output of the task python function is identical. I guess we need to > help bitbake > here. Any idea how? Own stamping logic? > > Jan ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v4 1/1] meta: added do_transform_template task as templating system and switch 2019-03-25 7:53 ` Claudius Heine @ 2019-03-25 8:13 ` Jan Kiszka 2019-03-25 8:21 ` Claudius Heine 0 siblings, 1 reply; 9+ messages in thread From: Jan Kiszka @ 2019-03-25 8:13 UTC (permalink / raw) To: Claudius Heine, isar-users; +Cc: Claudius Heine On 25.03.19 08:53, Claudius Heine wrote: > Hi Jan, > > On Sun, 2019-03-24 at 19:28 +0100, Jan Kiszka wrote: >> On 14.02.19 12:28, [ext] claudius.heine.ext@siemens.com wrote: >>> From: Claudius Heine <ch@denx.de> >>> >>> The new introduced variables are TEMPLATE_FILES and TEMPLATE_VARS. >>> TEMPLATE_FILES is a list of files that should be processed in the >>> `do_transform_template` task and TEMPLATE_VARS is a list of bitbake >>> variable names that should be substituted in the template file. >>> >> >> Bad news: This patch introduced a regression into the release (sorry, >> didn't >> manage to test earlier). It destroys the reuse of self-built >> identical packages >> for different targets. Namely, jailhouse-images now tries to build >> the shared >> (per arch) kernel multiple times, and that fails miserably because >> there is a >> common folder for all of them. The issue disappears when removing the >> do_transform_template task from the chain. > > Hmm, have you looked at the resulting 'TEMPLATE_VARS' variable? All > variables in there need to be the same in order for it to have the same > stamp. If that is not the case then maybe overwriting it instead of > appending it might work. > > These are the defaults set in 'template.bbclass': > > TEMPLATE_VARS ?= "PN PV DESCRIPTION HOMEPAGE MAINTAINER KERNEL_NAME > MACHINE DISTRO_ARCH" > > If I understand you correctly then your recipe should not be dependant > on the MACHINE? So either remove MACHINE 'TEMPLATE_VARS_remove = > "MACHINE"' or just set a 'TEMPLATE_VARS' that you really need in your > template files. OK, might be that "MACHINE" is the problem here. Will check. If that turns out to be true, it is highly counter-intuitive and should be fixed, at least for nop-jobs. Jan ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v4 1/1] meta: added do_transform_template task as templating system and switch 2019-03-25 8:13 ` Jan Kiszka @ 2019-03-25 8:21 ` Claudius Heine 2019-03-25 8:44 ` Jan Kiszka 0 siblings, 1 reply; 9+ messages in thread From: Claudius Heine @ 2019-03-25 8:21 UTC (permalink / raw) To: Jan Kiszka, isar-users; +Cc: Claudius Heine Hi, On 25/03/2019 09.13, Jan Kiszka wrote: > On 25.03.19 08:53, Claudius Heine wrote: >> Hi Jan, >> >> On Sun, 2019-03-24 at 19:28 +0100, Jan Kiszka wrote: >>> On 14.02.19 12:28, [ext] claudius.heine.ext@siemens.com wrote: >>>> From: Claudius Heine <ch@denx.de> >>>> >>>> The new introduced variables are TEMPLATE_FILES and TEMPLATE_VARS. >>>> TEMPLATE_FILES is a list of files that should be processed in the >>>> `do_transform_template` task and TEMPLATE_VARS is a list of bitbake >>>> variable names that should be substituted in the template file. >>>> >>> >>> Bad news: This patch introduced a regression into the release (sorry, >>> didn't >>> manage to test earlier). It destroys the reuse of self-built >>> identical packages >>> for different targets. Namely, jailhouse-images now tries to build >>> the shared >>> (per arch) kernel multiple times, and that fails miserably because >>> there is a >>> common folder for all of them. The issue disappears when removing the >>> do_transform_template task from the chain. >> >> Hmm, have you looked at the resulting 'TEMPLATE_VARS' variable? All >> variables in there need to be the same in order for it to have the same >> stamp. If that is not the case then maybe overwriting it instead of >> appending it might work. >> >> These are the defaults set in 'template.bbclass': >> >> TEMPLATE_VARS ?= "PN PV DESCRIPTION HOMEPAGE MAINTAINER KERNEL_NAME >> MACHINE DISTRO_ARCH" >> >> If I understand you correctly then your recipe should not be dependant >> on the MACHINE? So either remove MACHINE 'TEMPLATE_VARS_remove = >> "MACHINE"' or just set a 'TEMPLATE_VARS' that you really need in your >> template files. > > OK, might be that "MACHINE" is the problem here. Will check. If that > turns out > to be true, it is highly counter-intuitive and should be fixed, at least > for > nop-jobs. Well sane defaults is always WIP :) I think I added this because the u-boot recipe in meta needed it. It seems that u-boot is special here. Claudius -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v4 1/1] meta: added do_transform_template task as templating system and switch 2019-03-25 8:21 ` Claudius Heine @ 2019-03-25 8:44 ` Jan Kiszka 2019-03-25 11:08 ` Jan Kiszka 0 siblings, 1 reply; 9+ messages in thread From: Jan Kiszka @ 2019-03-25 8:44 UTC (permalink / raw) To: Claudius Heine, isar-users; +Cc: Claudius Heine On 25.03.19 09:21, Claudius Heine wrote: > Hi, > > On 25/03/2019 09.13, Jan Kiszka wrote: >> On 25.03.19 08:53, Claudius Heine wrote: >>> Hi Jan, >>> >>> On Sun, 2019-03-24 at 19:28 +0100, Jan Kiszka wrote: >>>> On 14.02.19 12:28, [ext] claudius.heine.ext@siemens.com wrote: >>>>> From: Claudius Heine <ch@denx.de> >>>>> >>>>> The new introduced variables are TEMPLATE_FILES and TEMPLATE_VARS. >>>>> TEMPLATE_FILES is a list of files that should be processed in the >>>>> `do_transform_template` task and TEMPLATE_VARS is a list of bitbake >>>>> variable names that should be substituted in the template file. >>>>> >>>> >>>> Bad news: This patch introduced a regression into the release (sorry, >>>> didn't >>>> manage to test earlier). It destroys the reuse of self-built >>>> identical packages >>>> for different targets. Namely, jailhouse-images now tries to build >>>> the shared >>>> (per arch) kernel multiple times, and that fails miserably because >>>> there is a >>>> common folder for all of them. The issue disappears when removing the >>>> do_transform_template task from the chain. >>> >>> Hmm, have you looked at the resulting 'TEMPLATE_VARS' variable? All >>> variables in there need to be the same in order for it to have the same >>> stamp. If that is not the case then maybe overwriting it instead of >>> appending it might work. >>> >>> These are the defaults set in 'template.bbclass': >>> >>> TEMPLATE_VARS ?= "PN PV DESCRIPTION HOMEPAGE MAINTAINER KERNEL_NAME >>> MACHINE DISTRO_ARCH" >>> >>> If I understand you correctly then your recipe should not be dependant >>> on the MACHINE? So either remove MACHINE 'TEMPLATE_VARS_remove = >>> "MACHINE"' or just set a 'TEMPLATE_VARS' that you really need in your >>> template files. >> >> OK, might be that "MACHINE" is the problem here. Will check. If that turns out >> to be true, it is highly counter-intuitive and should be fixed, at least for >> nop-jobs. > > Well sane defaults is always WIP :) > > I think I added this because the u-boot recipe in meta needed it. It seems that > u-boot is special here. OK, clearing TEMPLATE_VARS seems to address the issue (still seeing others, but that might be a local topic). Given these side effects, I'm in favor of removing MACHINE from the defaults. The other vars look harmless, at least so far. Jan ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v4 1/1] meta: added do_transform_template task as templating system and switch 2019-03-25 8:44 ` Jan Kiszka @ 2019-03-25 11:08 ` Jan Kiszka 0 siblings, 0 replies; 9+ messages in thread From: Jan Kiszka @ 2019-03-25 11:08 UTC (permalink / raw) To: Claudius Heine, isar-users; +Cc: Claudius Heine On 25.03.19 09:44, Jan Kiszka wrote: > On 25.03.19 09:21, Claudius Heine wrote: >> Hi, >> >> On 25/03/2019 09.13, Jan Kiszka wrote: >>> On 25.03.19 08:53, Claudius Heine wrote: >>>> Hi Jan, >>>> >>>> On Sun, 2019-03-24 at 19:28 +0100, Jan Kiszka wrote: >>>>> On 14.02.19 12:28, [ext] claudius.heine.ext@siemens.com wrote: >>>>>> From: Claudius Heine <ch@denx.de> >>>>>> >>>>>> The new introduced variables are TEMPLATE_FILES and TEMPLATE_VARS. >>>>>> TEMPLATE_FILES is a list of files that should be processed in the >>>>>> `do_transform_template` task and TEMPLATE_VARS is a list of bitbake >>>>>> variable names that should be substituted in the template file. >>>>>> >>>>> >>>>> Bad news: This patch introduced a regression into the release (sorry, >>>>> didn't >>>>> manage to test earlier). It destroys the reuse of self-built >>>>> identical packages >>>>> for different targets. Namely, jailhouse-images now tries to build >>>>> the shared >>>>> (per arch) kernel multiple times, and that fails miserably because >>>>> there is a >>>>> common folder for all of them. The issue disappears when removing the >>>>> do_transform_template task from the chain. >>>> >>>> Hmm, have you looked at the resulting 'TEMPLATE_VARS' variable? All >>>> variables in there need to be the same in order for it to have the same >>>> stamp. If that is not the case then maybe overwriting it instead of >>>> appending it might work. >>>> >>>> These are the defaults set in 'template.bbclass': >>>> >>>> TEMPLATE_VARS ?= "PN PV DESCRIPTION HOMEPAGE MAINTAINER KERNEL_NAME >>>> MACHINE DISTRO_ARCH" >>>> >>>> If I understand you correctly then your recipe should not be dependant >>>> on the MACHINE? So either remove MACHINE 'TEMPLATE_VARS_remove = >>>> "MACHINE"' or just set a 'TEMPLATE_VARS' that you really need in your >>>> template files. >>> >>> OK, might be that "MACHINE" is the problem here. Will check. If that turns out >>> to be true, it is highly counter-intuitive and should be fixed, at least for >>> nop-jobs. >> >> Well sane defaults is always WIP :) >> >> I think I added this because the u-boot recipe in meta needed it. It seems >> that u-boot is special here. > > OK, clearing TEMPLATE_VARS seems to address the issue (still seeing others, but > that might be a local topic). > > Given these side effects, I'm in favor of removing MACHINE from the defaults. > The other vars look harmless, at least so far. > Actually, we have to: Even standard packages like isar-cfg-localepurge start to fail building when there are multiple targets of the same arch pulling them in. They share the same workdir, thus break when trying to be built multiple times. Will send a patch. Jan ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v4 0/1] Template system 2019-02-14 11:28 [PATCH v4 0/1] Template system claudius.heine.ext 2019-02-14 11:28 ` [PATCH v4 1/1] meta: added do_transform_template task as templating system and switch claudius.heine.ext @ 2019-02-15 14:45 ` Maxim Yu. Osipov 1 sibling, 0 replies; 9+ messages in thread From: Maxim Yu. Osipov @ 2019-02-15 14:45 UTC (permalink / raw) To: claudius.heine.ext, isar-users; +Cc: Claudius Heine On 2/14/19 12:28 PM, claudius.heine.ext@siemens.com wrote: > From: Claudius Heine <ch@denx.de> > > Hi, > > apparently the '??=' in the template class prevented the '+=' in the > u-boot class to properly extend the TEMPLATE_VARS variable. I now > changed that to '?=' and that solved this issue. > > regards, > Claudius > > Changes from v3: > - replaced '??=' with '?=' in template.bbclass > > Changes from v2: > - put subprocess into context manager > - added 'gettext-base' isar dependency > - remove 'before do_build' from addtask > > Changes from v1: > - fixed spelling mistakes of documentation > > Claudius Heine (1): > meta: added do_transform_template task as templating system and switch Applied to the 'next', Thanks, Maxim. > doc/technical_overview.md | 25 ++++++++ > doc/user_manual.md | 1 + > meta/classes/base.bbclass | 1 + > meta/classes/dpkg-base.bbclass | 2 +- > meta/classes/template.bbclass | 62 +++++++++++++++++++ > .../debian/{changelog => changelog.tmpl} | 2 +- > meta/recipes-bsp/u-boot/files/debian/control | 19 ------ > .../u-boot/files/debian/control.tmpl | 19 ++++++ > meta/recipes-bsp/u-boot/u-boot-custom.inc | 12 ++-- > .../debian/{changelog => changelog.tmpl} | 2 +- > .../linux-module/files/debian/control | 11 ---- > .../linux-module/files/debian/control.tmpl | 11 ++++ > meta/recipes-kernel/linux-module/module.inc | 7 +-- > 13 files changed, 130 insertions(+), 44 deletions(-) > create mode 100644 meta/classes/template.bbclass > rename meta/recipes-bsp/u-boot/files/debian/{changelog => changelog.tmpl} (74%) > delete mode 100644 meta/recipes-bsp/u-boot/files/debian/control > create mode 100644 meta/recipes-bsp/u-boot/files/debian/control.tmpl > rename meta/recipes-kernel/linux-module/files/debian/{changelog => changelog.tmpl} (74%) > delete mode 100644 meta/recipes-kernel/linux-module/files/debian/control > create mode 100644 meta/recipes-kernel/linux-module/files/debian/control.tmpl > -- Maxim Osipov ilbers GmbH Maria-Merian-Str. 8 85521 Ottobrunn Germany +49 (151) 6517 6917 mosipov@ilbers.de http://ilbers.de/ Commercial register Munich, HRB 214197 General Manager: Baurzhan Ismagulov ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2019-03-25 11:08 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-02-14 11:28 [PATCH v4 0/1] Template system claudius.heine.ext 2019-02-14 11:28 ` [PATCH v4 1/1] meta: added do_transform_template task as templating system and switch claudius.heine.ext 2019-03-24 18:28 ` Jan Kiszka 2019-03-25 7:53 ` Claudius Heine 2019-03-25 8:13 ` Jan Kiszka 2019-03-25 8:21 ` Claudius Heine 2019-03-25 8:44 ` Jan Kiszka 2019-03-25 11:08 ` Jan Kiszka 2019-02-15 14:45 ` [PATCH v4 0/1] Template system Maxim Yu. Osipov
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox