* [PATCH v4 0/1] Template system
@ 2019-02-14 11:28 claudius.heine.ext
2019-02-14 11:28 ` [PATCH v4 1/1] meta: added do_transform_template task as templating system and switch claudius.heine.ext
2019-02-15 14:45 ` [PATCH v4 0/1] Template system Maxim Yu. Osipov
0 siblings, 2 replies; 9+ messages in thread
From: claudius.heine.ext @ 2019-02-14 11:28 UTC (permalink / raw)
To: isar-users; +Cc: Claudius Heine
From: Claudius Heine <ch@denx.de>
Hi,
apparently the '??=' in the template class prevented the '+=' in the
u-boot class to properly extend the TEMPLATE_VARS variable. I now
changed that to '?=' and that solved this issue.
regards,
Claudius
Changes from v3:
- replaced '??=' with '?=' in template.bbclass
Changes from v2:
- put subprocess into context manager
- added 'gettext-base' isar dependency
- remove 'before do_build' from addtask
Changes from v1:
- fixed spelling mistakes of documentation
Claudius Heine (1):
meta: added do_transform_template task as templating system and switch
doc/technical_overview.md | 25 ++++++++
doc/user_manual.md | 1 +
meta/classes/base.bbclass | 1 +
meta/classes/dpkg-base.bbclass | 2 +-
meta/classes/template.bbclass | 62 +++++++++++++++++++
.../debian/{changelog => changelog.tmpl} | 2 +-
meta/recipes-bsp/u-boot/files/debian/control | 19 ------
.../u-boot/files/debian/control.tmpl | 19 ++++++
meta/recipes-bsp/u-boot/u-boot-custom.inc | 12 ++--
.../debian/{changelog => changelog.tmpl} | 2 +-
.../linux-module/files/debian/control | 11 ----
.../linux-module/files/debian/control.tmpl | 11 ++++
meta/recipes-kernel/linux-module/module.inc | 7 +--
13 files changed, 130 insertions(+), 44 deletions(-)
create mode 100644 meta/classes/template.bbclass
rename meta/recipes-bsp/u-boot/files/debian/{changelog => changelog.tmpl} (74%)
delete mode 100644 meta/recipes-bsp/u-boot/files/debian/control
create mode 100644 meta/recipes-bsp/u-boot/files/debian/control.tmpl
rename meta/recipes-kernel/linux-module/files/debian/{changelog => changelog.tmpl} (74%)
delete mode 100644 meta/recipes-kernel/linux-module/files/debian/control
create mode 100644 meta/recipes-kernel/linux-module/files/debian/control.tmpl
--
2.20.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH v4 1/1] meta: added do_transform_template task as templating system and switch
2019-02-14 11:28 [PATCH v4 0/1] Template system claudius.heine.ext
@ 2019-02-14 11:28 ` claudius.heine.ext
2019-03-24 18:28 ` Jan Kiszka
2019-02-15 14:45 ` [PATCH v4 0/1] Template system Maxim Yu. Osipov
1 sibling, 1 reply; 9+ messages in thread
From: claudius.heine.ext @ 2019-02-14 11:28 UTC (permalink / raw)
To: isar-users; +Cc: Claudius Heine
From: Claudius Heine <ch@denx.de>
The new introduced variables are TEMPLATE_FILES and TEMPLATE_VARS.
TEMPLATE_FILES is a list of files that should be processed in the
`do_transform_template` task and TEMPLATE_VARS is a list of bitbake
variable names that should be substituted in the template file.
Signed-off-by: Claudius Heine <ch@denx.de>
---
doc/technical_overview.md | 25 ++++++++
doc/user_manual.md | 1 +
meta/classes/base.bbclass | 1 +
meta/classes/dpkg-base.bbclass | 2 +-
meta/classes/template.bbclass | 62 +++++++++++++++++++
.../debian/{changelog => changelog.tmpl} | 2 +-
meta/recipes-bsp/u-boot/files/debian/control | 19 ------
.../u-boot/files/debian/control.tmpl | 19 ++++++
meta/recipes-bsp/u-boot/u-boot-custom.inc | 12 ++--
.../debian/{changelog => changelog.tmpl} | 2 +-
.../linux-module/files/debian/control | 11 ----
.../linux-module/files/debian/control.tmpl | 11 ++++
meta/recipes-kernel/linux-module/module.inc | 7 +--
13 files changed, 130 insertions(+), 44 deletions(-)
create mode 100644 meta/classes/template.bbclass
rename meta/recipes-bsp/u-boot/files/debian/{changelog => changelog.tmpl} (74%)
delete mode 100644 meta/recipes-bsp/u-boot/files/debian/control
create mode 100644 meta/recipes-bsp/u-boot/files/debian/control.tmpl
rename meta/recipes-kernel/linux-module/files/debian/{changelog => changelog.tmpl} (74%)
delete mode 100644 meta/recipes-kernel/linux-module/files/debian/control
create mode 100644 meta/recipes-kernel/linux-module/files/debian/control.tmpl
diff --git a/doc/technical_overview.md b/doc/technical_overview.md
index 7efbeee..0ddcabc 100644
--- a/doc/technical_overview.md
+++ b/doc/technical_overview.md
@@ -283,3 +283,28 @@ cross-compilation, this is done by the same tools as for native compilation.
Depending on ISAR_CROSS_COMPILE value, additional architecture specifiers are
passed to build tools automatically, so this is absolutely transparent from
the user point of view.
+
+## 3.9 Additional features
+
+### 3.9.1 Template files
+
+A basic templating system implemented on top `envsubst(1)` is available. It
+allows limited access to bitbake variables for file generation.
+
+This system is implemented in the `template.bbclass` and defines the
+`do_transform_template` task, the `TEMPLATE_FILES` and `TEMPLATE_VARS`
+variables. Tasks that use need to use files generated via templates need
+to be executed after the `do_transform_template` task.
+
+The `TEMPLATE_FILES` variable contains a space separated list of template
+files and the `TEMPLATE_VARS` variable a space separated list of bitbake
+variable names. The `do_transform_template` task takes these and generates
+files using `envsubst(1)`. The output files are placed in the same directory
+as the template files. The name of the output files are either the name of the
+template file with `.tmpl` removed from the file name end or, if the template
+file did not end with `.tmpl`, the name of the template file with `.out` added
+to the end.
+
+Only template files from the `WORKDIR` are accepted. Either specify relative
+paths based on the recipes `WORKDIR` or absolute paths containing the `WORKDIR`
+in the `TEMPLATE_FILES` variable.
diff --git a/doc/user_manual.md b/doc/user_manual.md
index ff4939c..c1b154f 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -68,6 +68,7 @@ qemu-user-static
binfmt-support
sudo
reprepro
+gettext-base
```
Notes:
diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 6ca93bf..4279a68 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -25,6 +25,7 @@ die() {
}
inherit logging
+inherit template
# Derived from bitbake: bitbake/classes/base.bbclass
addtask showdata
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 175dc80..320869b 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -60,7 +60,7 @@ do_prepare_build() {
true
}
-addtask prepare_build after do_patch before do_build
+addtask prepare_build after do_patch do_transform_template before do_build
do_prepare_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
# If Isar recipes depend on each other, they typically need the package
# deployed to isar-apt
diff --git a/meta/classes/template.bbclass b/meta/classes/template.bbclass
new file mode 100644
index 0000000..324511a
--- /dev/null
+++ b/meta/classes/template.bbclass
@@ -0,0 +1,62 @@
+# This software is a part of ISAR.
+# Copyright (C) Siemens AG, 2019
+#
+# SPDX-License-Identifier: MIT
+
+TEMPLATE_FILES ?= ""
+TEMPLATE_VARS ?= "PN PV DESCRIPTION HOMEPAGE MAINTAINER KERNEL_NAME MACHINE \
+ DISTRO_ARCH"
+
+do_transform_template[vardeps] = "TEMPLATE_FILES ${TEMPLATE_VARS}"
+do_transform_template[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+python do_transform_template() {
+ import subprocess, contextlib
+
+ workdir = os.path.normpath(d.getVar('WORKDIR', True))
+
+ template_vars = (d.getVar('TEMPLATE_VARS', True) or "").split()
+ if len(template_vars) == 0:
+ return
+
+ template_files = (d.getVar('TEMPLATE_FILES', True) or "").split()
+ if len(template_files) == 0:
+ return
+
+ cmd = "envsubst"
+ args = " ".join("\${{{}}}".format(i) for i in template_vars)
+
+ # Copy current process environment and add template variables
+ # from bitbake data store:
+ env = os.environ.copy()
+ for varname in template_vars:
+ value = d.getVar(varname, True)
+ if value:
+ env.update({varname: value})
+
+ for template_file in template_files:
+ # Normpath and workdir checks should prevent accidential or
+ # uninformed changes to files outside of the tmp and workdirectoy
+ template_file = os.path.normpath(template_file)
+
+ # Convert relative paths to absolut paths based on the workdir:
+ if not os.path.isabs(template_file):
+ template_file = os.path.normpath(os.path.join(workdir, template_file))
+
+ if not template_file.startswith(workdir):
+ bb.fatal("Template file ({}) is not within workdir ({})"
+ .format(template_file, workdir))
+
+ output_file = (os.path.splitext(template_file)[0]
+ if template_file.endswith(".tmpl")
+ else (template_file + ".out"))
+ bb.note("{} {} [in: {} out: {}]".format(cmd, args,
+ template_file, output_file))
+ with contextlib.ExitStack() as stack:
+ input = stack.enter_context(open(template_file, 'rb'))
+ output = stack.enter_context(open(output_file, 'wb'))
+ process = stack.enter_context(subprocess.Popen([cmd, args], stdin=input,
+ stdout=output, env=env))
+ if process.wait() != 0:
+ bb.fatal("processing of template failed")
+}
+addtask do_transform_template after do_unpack
diff --git a/meta/recipes-bsp/u-boot/files/debian/changelog b/meta/recipes-bsp/u-boot/files/debian/changelog.tmpl
similarity index 74%
rename from meta/recipes-bsp/u-boot/files/debian/changelog
rename to meta/recipes-bsp/u-boot/files/debian/changelog.tmpl
index c1c3516..6e59e06 100644
--- a/meta/recipes-bsp/u-boot/files/debian/changelog
+++ b/meta/recipes-bsp/u-boot/files/debian/changelog.tmpl
@@ -1,4 +1,4 @@
-@PN@ (@PV@) unstable; urgency=low
+${PN} (${PV}) unstable; urgency=low
* Generated package.
diff --git a/meta/recipes-bsp/u-boot/files/debian/control b/meta/recipes-bsp/u-boot/files/debian/control
deleted file mode 100644
index 6b4c839..0000000
--- a/meta/recipes-bsp/u-boot/files/debian/control
+++ /dev/null
@@ -1,19 +0,0 @@
-Source: @PN@
-Section: admin
-Priority: optional
-Standards-Version: 3.9.6
-Build-Depends: @BUILD_DEPENDS@
-Maintainer: ISAR project <isar-users@googlegroups.com>
-
-Package: u-boot-@MACHINE@
-Architecture: @DISTRO_ARCH@
-Description: @DESCRIPTION@, bootloader binaries
-
-Package: u-boot-@MACHINE@-dev
-Architecture: @DISTRO_ARCH@
-Description: @DESCRIPTION@, bootloader libraries
-
-Package: u-boot-tools
-Architecture: linux-any
-Depends: ${shlibs:Depends}, ${misc:Depends}
-Description: @DESCRIPTION@, companion tools
diff --git a/meta/recipes-bsp/u-boot/files/debian/control.tmpl b/meta/recipes-bsp/u-boot/files/debian/control.tmpl
new file mode 100644
index 0000000..5c1cc92
--- /dev/null
+++ b/meta/recipes-bsp/u-boot/files/debian/control.tmpl
@@ -0,0 +1,19 @@
+Source: ${PN}
+Section: admin
+Priority: optional
+Standards-Version: 3.9.6
+Build-Depends: ${BUILD_DEPENDS}
+Maintainer: ISAR project <isar-users@googlegroups.com>
+
+Package: u-boot-${MACHINE}
+Architecture: ${DISTRO_ARCH}
+Description: ${DESCRIPTION}, bootloader binaries
+
+Package: u-boot-${MACHINE}-dev
+Architecture: ${DISTRO_ARCH}
+Description: ${DESCRIPTION}, bootloader libraries
+
+Package: u-boot-tools
+Architecture: linux-any
+Depends: ${shlibs:Depends}, ${misc:Depends}
+Description: ${DESCRIPTION}, companion tools
diff --git a/meta/recipes-bsp/u-boot/u-boot-custom.inc b/meta/recipes-bsp/u-boot/u-boot-custom.inc
index 4b38c88..c6aab43 100644
--- a/meta/recipes-bsp/u-boot/u-boot-custom.inc
+++ b/meta/recipes-bsp/u-boot/u-boot-custom.inc
@@ -15,16 +15,14 @@ inherit dpkg
SRC_URI += "file://debian/"
-U_BOOT_BUILD_DEPENDS ?= "bc, bison, flex, device-tree-compiler"
+BUILD_DEPENDS ?= "bc, bison, flex, device-tree-compiler"
+
+TEMPLATE_FILES = "debian/changelog.tmpl \
+ debian/control.tmpl"
+TEMPLATE_VARS += "BUILD_DEPENDS"
do_prepare_build() {
cp -r ${WORKDIR}/debian ${S}/
- sed -i -e 's/@PN@/${PN}/g' -e 's/@PV@/${PV}/g' \
- -e 's/@BUILD_DEPENDS@/${U_BOOT_BUILD_DEPENDS}/g' \
- -e 's/@MACHINE@/${MACHINE}/g' \
- -e 's/@DISTRO_ARCH@/${DISTRO_ARCH}/g' \
- -e 's/@DESCRIPTION@/${DESCRIPTION}/g' \
- ${S}/debian/changelog ${S}/debian/control
echo "${U_BOOT_BIN} /usr/lib/u-boot/${MACHINE}" > \
${S}/debian/u-boot-${MACHINE}.install
diff --git a/meta/recipes-kernel/linux-module/files/debian/changelog b/meta/recipes-kernel/linux-module/files/debian/changelog.tmpl
similarity index 74%
rename from meta/recipes-kernel/linux-module/files/debian/changelog
rename to meta/recipes-kernel/linux-module/files/debian/changelog.tmpl
index c1c3516..6e59e06 100644
--- a/meta/recipes-kernel/linux-module/files/debian/changelog
+++ b/meta/recipes-kernel/linux-module/files/debian/changelog.tmpl
@@ -1,4 +1,4 @@
-@PN@ (@PV@) unstable; urgency=low
+${PN} (${PV}) unstable; urgency=low
* Generated package.
diff --git a/meta/recipes-kernel/linux-module/files/debian/control b/meta/recipes-kernel/linux-module/files/debian/control
deleted file mode 100644
index 1ee634c..0000000
--- a/meta/recipes-kernel/linux-module/files/debian/control
+++ /dev/null
@@ -1,11 +0,0 @@
-Source: @PN@
-Section: kernel
-Priority: optional
-Standards-Version: 3.9.6
-Build-Depends: linux-headers-@KERNEL_NAME@
-Maintainer: ISAR project <isar-users@googlegroups.com>
-
-Package: @PN@
-Architecture: any
-Depends: linux-image-@KERNEL_NAME@, kmod
-Description: @DESCRIPTION@
diff --git a/meta/recipes-kernel/linux-module/files/debian/control.tmpl b/meta/recipes-kernel/linux-module/files/debian/control.tmpl
new file mode 100644
index 0000000..3b3292d
--- /dev/null
+++ b/meta/recipes-kernel/linux-module/files/debian/control.tmpl
@@ -0,0 +1,11 @@
+Source: ${PN}
+Section: kernel
+Priority: optional
+Standards-Version: 3.9.6
+Build-Depends: linux-headers-${KERNEL_NAME}
+Maintainer: ISAR project <isar-users@googlegroups.com>
+
+Package: ${PN}
+Architecture: any
+Depends: linux-image-${KERNEL_NAME}, kmod
+Description: ${DESCRIPTION}
diff --git a/meta/recipes-kernel/linux-module/module.inc b/meta/recipes-kernel/linux-module/module.inc
index cb7b8ad..44edd58 100644
--- a/meta/recipes-kernel/linux-module/module.inc
+++ b/meta/recipes-kernel/linux-module/module.inc
@@ -19,12 +19,11 @@ AUTOLOAD ?= ""
inherit dpkg
+TEMPLATE_FILES = "debian/control.tmpl \
+ debian/changelog.tmpl"
+
do_prepare_build() {
cp -r ${WORKDIR}/debian ${S}/
- sed -i -e 's/@PN@/${PN}/g' -e 's/@PV@/${PV}/g' \
- -e 's/@KERNEL_NAME@/${KERNEL_NAME}/g' \
- -e 's/@DESCRIPTION@/${DESCRIPTION}/g' \
- ${S}/debian/changelog ${S}/debian/control
for module in "${AUTOLOAD}"; do
echo "echo $module >> /etc/modules" >> ${S}/debian/postinst
--
2.20.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v4 0/1] Template system
2019-02-14 11:28 [PATCH v4 0/1] Template system claudius.heine.ext
2019-02-14 11:28 ` [PATCH v4 1/1] meta: added do_transform_template task as templating system and switch claudius.heine.ext
@ 2019-02-15 14:45 ` Maxim Yu. Osipov
1 sibling, 0 replies; 9+ messages in thread
From: Maxim Yu. Osipov @ 2019-02-15 14:45 UTC (permalink / raw)
To: claudius.heine.ext, isar-users; +Cc: Claudius Heine
On 2/14/19 12:28 PM, claudius.heine.ext@siemens.com wrote:
> From: Claudius Heine <ch@denx.de>
>
> Hi,
>
> apparently the '??=' in the template class prevented the '+=' in the
> u-boot class to properly extend the TEMPLATE_VARS variable. I now
> changed that to '?=' and that solved this issue.
>
> regards,
> Claudius
>
> Changes from v3:
> - replaced '??=' with '?=' in template.bbclass
>
> Changes from v2:
> - put subprocess into context manager
> - added 'gettext-base' isar dependency
> - remove 'before do_build' from addtask
>
> Changes from v1:
> - fixed spelling mistakes of documentation
>
> Claudius Heine (1):
> meta: added do_transform_template task as templating system and switch
Applied to the 'next',
Thanks,
Maxim.
> doc/technical_overview.md | 25 ++++++++
> doc/user_manual.md | 1 +
> meta/classes/base.bbclass | 1 +
> meta/classes/dpkg-base.bbclass | 2 +-
> meta/classes/template.bbclass | 62 +++++++++++++++++++
> .../debian/{changelog => changelog.tmpl} | 2 +-
> meta/recipes-bsp/u-boot/files/debian/control | 19 ------
> .../u-boot/files/debian/control.tmpl | 19 ++++++
> meta/recipes-bsp/u-boot/u-boot-custom.inc | 12 ++--
> .../debian/{changelog => changelog.tmpl} | 2 +-
> .../linux-module/files/debian/control | 11 ----
> .../linux-module/files/debian/control.tmpl | 11 ++++
> meta/recipes-kernel/linux-module/module.inc | 7 +--
> 13 files changed, 130 insertions(+), 44 deletions(-)
> create mode 100644 meta/classes/template.bbclass
> rename meta/recipes-bsp/u-boot/files/debian/{changelog => changelog.tmpl} (74%)
> delete mode 100644 meta/recipes-bsp/u-boot/files/debian/control
> create mode 100644 meta/recipes-bsp/u-boot/files/debian/control.tmpl
> rename meta/recipes-kernel/linux-module/files/debian/{changelog => changelog.tmpl} (74%)
> delete mode 100644 meta/recipes-kernel/linux-module/files/debian/control
> create mode 100644 meta/recipes-kernel/linux-module/files/debian/control.tmpl
>
--
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v4 1/1] meta: added do_transform_template task as templating system and switch
2019-02-14 11:28 ` [PATCH v4 1/1] meta: added do_transform_template task as templating system and switch claudius.heine.ext
@ 2019-03-24 18:28 ` Jan Kiszka
2019-03-25 7:53 ` Claudius Heine
0 siblings, 1 reply; 9+ messages in thread
From: Jan Kiszka @ 2019-03-24 18:28 UTC (permalink / raw)
To: [ext] claudius.heine.ext@siemens.com, isar-users; +Cc: Claudius Heine
On 14.02.19 12:28, [ext] claudius.heine.ext@siemens.com wrote:
> From: Claudius Heine <ch@denx.de>
>
> The new introduced variables are TEMPLATE_FILES and TEMPLATE_VARS.
> TEMPLATE_FILES is a list of files that should be processed in the
> `do_transform_template` task and TEMPLATE_VARS is a list of bitbake
> variable names that should be substituted in the template file.
>
Bad news: This patch introduced a regression into the release (sorry, didn't
manage to test earlier). It destroys the reuse of self-built identical packages
for different targets. Namely, jailhouse-images now tries to build the shared
(per arch) kernel multiple times, and that fails miserably because there is a
common folder for all of them. The issue disappears when removing the
do_transform_template task from the chain.
Digging deeper, the reason is that do_transform_template is not shared but run
multiple times (even if nothing has to be done), forcing all succeeding tasks
into private mode as well. I suspect that's because bitbake cannot assess if the
output of the task python function is identical. I guess we need to help bitbake
here. Any idea how? Own stamping logic?
Jan
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v4 1/1] meta: added do_transform_template task as templating system and switch
2019-03-24 18:28 ` Jan Kiszka
@ 2019-03-25 7:53 ` Claudius Heine
2019-03-25 8:13 ` Jan Kiszka
0 siblings, 1 reply; 9+ messages in thread
From: Claudius Heine @ 2019-03-25 7:53 UTC (permalink / raw)
To: Jan Kiszka, isar-users; +Cc: Claudius Heine
Hi Jan,
On Sun, 2019-03-24 at 19:28 +0100, Jan Kiszka wrote:
> On 14.02.19 12:28, [ext] claudius.heine.ext@siemens.com wrote:
> > From: Claudius Heine <ch@denx.de>
> >
> > The new introduced variables are TEMPLATE_FILES and TEMPLATE_VARS.
> > TEMPLATE_FILES is a list of files that should be processed in the
> > `do_transform_template` task and TEMPLATE_VARS is a list of bitbake
> > variable names that should be substituted in the template file.
> >
>
> Bad news: This patch introduced a regression into the release (sorry,
> didn't
> manage to test earlier). It destroys the reuse of self-built
> identical packages
> for different targets. Namely, jailhouse-images now tries to build
> the shared
> (per arch) kernel multiple times, and that fails miserably because
> there is a
> common folder for all of them. The issue disappears when removing the
> do_transform_template task from the chain.
Hmm, have you looked at the resulting 'TEMPLATE_VARS' variable? All
variables in there need to be the same in order for it to have the same
stamp. If that is not the case then maybe overwriting it instead of
appending it might work.
These are the defaults set in 'template.bbclass':
TEMPLATE_VARS ?= "PN PV DESCRIPTION HOMEPAGE MAINTAINER KERNEL_NAME
MACHINE DISTRO_ARCH"
If I understand you correctly then your recipe should not be dependant
on the MACHINE? So either remove MACHINE 'TEMPLATE_VARS_remove =
"MACHINE"' or just set a 'TEMPLATE_VARS' that you really need in your
template files.
Cheers,
Claudius
>
> Digging deeper, the reason is that do_transform_template is not
> shared but run
> multiple times (even if nothing has to be done), forcing all
> succeeding tasks
> into private mode as well. I suspect that's because bitbake cannot
> assess if the
> output of the task python function is identical. I guess we need to
> help bitbake
> here. Any idea how? Own stamping logic?
>
> Jan
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v4 1/1] meta: added do_transform_template task as templating system and switch
2019-03-25 7:53 ` Claudius Heine
@ 2019-03-25 8:13 ` Jan Kiszka
2019-03-25 8:21 ` Claudius Heine
0 siblings, 1 reply; 9+ messages in thread
From: Jan Kiszka @ 2019-03-25 8:13 UTC (permalink / raw)
To: Claudius Heine, isar-users; +Cc: Claudius Heine
On 25.03.19 08:53, Claudius Heine wrote:
> Hi Jan,
>
> On Sun, 2019-03-24 at 19:28 +0100, Jan Kiszka wrote:
>> On 14.02.19 12:28, [ext] claudius.heine.ext@siemens.com wrote:
>>> From: Claudius Heine <ch@denx.de>
>>>
>>> The new introduced variables are TEMPLATE_FILES and TEMPLATE_VARS.
>>> TEMPLATE_FILES is a list of files that should be processed in the
>>> `do_transform_template` task and TEMPLATE_VARS is a list of bitbake
>>> variable names that should be substituted in the template file.
>>>
>>
>> Bad news: This patch introduced a regression into the release (sorry,
>> didn't
>> manage to test earlier). It destroys the reuse of self-built
>> identical packages
>> for different targets. Namely, jailhouse-images now tries to build
>> the shared
>> (per arch) kernel multiple times, and that fails miserably because
>> there is a
>> common folder for all of them. The issue disappears when removing the
>> do_transform_template task from the chain.
>
> Hmm, have you looked at the resulting 'TEMPLATE_VARS' variable? All
> variables in there need to be the same in order for it to have the same
> stamp. If that is not the case then maybe overwriting it instead of
> appending it might work.
>
> These are the defaults set in 'template.bbclass':
>
> TEMPLATE_VARS ?= "PN PV DESCRIPTION HOMEPAGE MAINTAINER KERNEL_NAME
> MACHINE DISTRO_ARCH"
>
> If I understand you correctly then your recipe should not be dependant
> on the MACHINE? So either remove MACHINE 'TEMPLATE_VARS_remove =
> "MACHINE"' or just set a 'TEMPLATE_VARS' that you really need in your
> template files.
OK, might be that "MACHINE" is the problem here. Will check. If that turns out
to be true, it is highly counter-intuitive and should be fixed, at least for
nop-jobs.
Jan
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v4 1/1] meta: added do_transform_template task as templating system and switch
2019-03-25 8:13 ` Jan Kiszka
@ 2019-03-25 8:21 ` Claudius Heine
2019-03-25 8:44 ` Jan Kiszka
0 siblings, 1 reply; 9+ messages in thread
From: Claudius Heine @ 2019-03-25 8:21 UTC (permalink / raw)
To: Jan Kiszka, isar-users; +Cc: Claudius Heine
Hi,
On 25/03/2019 09.13, Jan Kiszka wrote:
> On 25.03.19 08:53, Claudius Heine wrote:
>> Hi Jan,
>>
>> On Sun, 2019-03-24 at 19:28 +0100, Jan Kiszka wrote:
>>> On 14.02.19 12:28, [ext] claudius.heine.ext@siemens.com wrote:
>>>> From: Claudius Heine <ch@denx.de>
>>>>
>>>> The new introduced variables are TEMPLATE_FILES and TEMPLATE_VARS.
>>>> TEMPLATE_FILES is a list of files that should be processed in the
>>>> `do_transform_template` task and TEMPLATE_VARS is a list of bitbake
>>>> variable names that should be substituted in the template file.
>>>>
>>>
>>> Bad news: This patch introduced a regression into the release (sorry,
>>> didn't
>>> manage to test earlier). It destroys the reuse of self-built
>>> identical packages
>>> for different targets. Namely, jailhouse-images now tries to build
>>> the shared
>>> (per arch) kernel multiple times, and that fails miserably because
>>> there is a
>>> common folder for all of them. The issue disappears when removing the
>>> do_transform_template task from the chain.
>>
>> Hmm, have you looked at the resulting 'TEMPLATE_VARS' variable? All
>> variables in there need to be the same in order for it to have the same
>> stamp. If that is not the case then maybe overwriting it instead of
>> appending it might work.
>>
>> These are the defaults set in 'template.bbclass':
>>
>> TEMPLATE_VARS ?= "PN PV DESCRIPTION HOMEPAGE MAINTAINER KERNEL_NAME
>> MACHINE DISTRO_ARCH"
>>
>> If I understand you correctly then your recipe should not be dependant
>> on the MACHINE? So either remove MACHINE 'TEMPLATE_VARS_remove =
>> "MACHINE"' or just set a 'TEMPLATE_VARS' that you really need in your
>> template files.
>
> OK, might be that "MACHINE" is the problem here. Will check. If that
> turns out
> to be true, it is highly counter-intuitive and should be fixed, at least
> for
> nop-jobs.
Well sane defaults is always WIP :)
I think I added this because the u-boot recipe in meta needed it. It
seems that u-boot is special here.
Claudius
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v4 1/1] meta: added do_transform_template task as templating system and switch
2019-03-25 8:21 ` Claudius Heine
@ 2019-03-25 8:44 ` Jan Kiszka
2019-03-25 11:08 ` Jan Kiszka
0 siblings, 1 reply; 9+ messages in thread
From: Jan Kiszka @ 2019-03-25 8:44 UTC (permalink / raw)
To: Claudius Heine, isar-users; +Cc: Claudius Heine
On 25.03.19 09:21, Claudius Heine wrote:
> Hi,
>
> On 25/03/2019 09.13, Jan Kiszka wrote:
>> On 25.03.19 08:53, Claudius Heine wrote:
>>> Hi Jan,
>>>
>>> On Sun, 2019-03-24 at 19:28 +0100, Jan Kiszka wrote:
>>>> On 14.02.19 12:28, [ext] claudius.heine.ext@siemens.com wrote:
>>>>> From: Claudius Heine <ch@denx.de>
>>>>>
>>>>> The new introduced variables are TEMPLATE_FILES and TEMPLATE_VARS.
>>>>> TEMPLATE_FILES is a list of files that should be processed in the
>>>>> `do_transform_template` task and TEMPLATE_VARS is a list of bitbake
>>>>> variable names that should be substituted in the template file.
>>>>>
>>>>
>>>> Bad news: This patch introduced a regression into the release (sorry,
>>>> didn't
>>>> manage to test earlier). It destroys the reuse of self-built
>>>> identical packages
>>>> for different targets. Namely, jailhouse-images now tries to build
>>>> the shared
>>>> (per arch) kernel multiple times, and that fails miserably because
>>>> there is a
>>>> common folder for all of them. The issue disappears when removing the
>>>> do_transform_template task from the chain.
>>>
>>> Hmm, have you looked at the resulting 'TEMPLATE_VARS' variable? All
>>> variables in there need to be the same in order for it to have the same
>>> stamp. If that is not the case then maybe overwriting it instead of
>>> appending it might work.
>>>
>>> These are the defaults set in 'template.bbclass':
>>>
>>> TEMPLATE_VARS ?= "PN PV DESCRIPTION HOMEPAGE MAINTAINER KERNEL_NAME
>>> MACHINE DISTRO_ARCH"
>>>
>>> If I understand you correctly then your recipe should not be dependant
>>> on the MACHINE? So either remove MACHINE 'TEMPLATE_VARS_remove =
>>> "MACHINE"' or just set a 'TEMPLATE_VARS' that you really need in your
>>> template files.
>>
>> OK, might be that "MACHINE" is the problem here. Will check. If that turns out
>> to be true, it is highly counter-intuitive and should be fixed, at least for
>> nop-jobs.
>
> Well sane defaults is always WIP :)
>
> I think I added this because the u-boot recipe in meta needed it. It seems that
> u-boot is special here.
OK, clearing TEMPLATE_VARS seems to address the issue (still seeing others, but
that might be a local topic).
Given these side effects, I'm in favor of removing MACHINE from the defaults.
The other vars look harmless, at least so far.
Jan
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v4 1/1] meta: added do_transform_template task as templating system and switch
2019-03-25 8:44 ` Jan Kiszka
@ 2019-03-25 11:08 ` Jan Kiszka
0 siblings, 0 replies; 9+ messages in thread
From: Jan Kiszka @ 2019-03-25 11:08 UTC (permalink / raw)
To: Claudius Heine, isar-users; +Cc: Claudius Heine
On 25.03.19 09:44, Jan Kiszka wrote:
> On 25.03.19 09:21, Claudius Heine wrote:
>> Hi,
>>
>> On 25/03/2019 09.13, Jan Kiszka wrote:
>>> On 25.03.19 08:53, Claudius Heine wrote:
>>>> Hi Jan,
>>>>
>>>> On Sun, 2019-03-24 at 19:28 +0100, Jan Kiszka wrote:
>>>>> On 14.02.19 12:28, [ext] claudius.heine.ext@siemens.com wrote:
>>>>>> From: Claudius Heine <ch@denx.de>
>>>>>>
>>>>>> The new introduced variables are TEMPLATE_FILES and TEMPLATE_VARS.
>>>>>> TEMPLATE_FILES is a list of files that should be processed in the
>>>>>> `do_transform_template` task and TEMPLATE_VARS is a list of bitbake
>>>>>> variable names that should be substituted in the template file.
>>>>>>
>>>>>
>>>>> Bad news: This patch introduced a regression into the release (sorry,
>>>>> didn't
>>>>> manage to test earlier). It destroys the reuse of self-built
>>>>> identical packages
>>>>> for different targets. Namely, jailhouse-images now tries to build
>>>>> the shared
>>>>> (per arch) kernel multiple times, and that fails miserably because
>>>>> there is a
>>>>> common folder for all of them. The issue disappears when removing the
>>>>> do_transform_template task from the chain.
>>>>
>>>> Hmm, have you looked at the resulting 'TEMPLATE_VARS' variable? All
>>>> variables in there need to be the same in order for it to have the same
>>>> stamp. If that is not the case then maybe overwriting it instead of
>>>> appending it might work.
>>>>
>>>> These are the defaults set in 'template.bbclass':
>>>>
>>>> TEMPLATE_VARS ?= "PN PV DESCRIPTION HOMEPAGE MAINTAINER KERNEL_NAME
>>>> MACHINE DISTRO_ARCH"
>>>>
>>>> If I understand you correctly then your recipe should not be dependant
>>>> on the MACHINE? So either remove MACHINE 'TEMPLATE_VARS_remove =
>>>> "MACHINE"' or just set a 'TEMPLATE_VARS' that you really need in your
>>>> template files.
>>>
>>> OK, might be that "MACHINE" is the problem here. Will check. If that turns out
>>> to be true, it is highly counter-intuitive and should be fixed, at least for
>>> nop-jobs.
>>
>> Well sane defaults is always WIP :)
>>
>> I think I added this because the u-boot recipe in meta needed it. It seems
>> that u-boot is special here.
>
> OK, clearing TEMPLATE_VARS seems to address the issue (still seeing others, but
> that might be a local topic).
>
> Given these side effects, I'm in favor of removing MACHINE from the defaults.
> The other vars look harmless, at least so far.
>
Actually, we have to: Even standard packages like isar-cfg-localepurge start to
fail building when there are multiple targets of the same arch pulling them in.
They share the same workdir, thus break when trying to be built multiple times.
Will send a patch.
Jan
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2019-03-25 11:08 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-02-14 11:28 [PATCH v4 0/1] Template system claudius.heine.ext
2019-02-14 11:28 ` [PATCH v4 1/1] meta: added do_transform_template task as templating system and switch claudius.heine.ext
2019-03-24 18:28 ` Jan Kiszka
2019-03-25 7:53 ` Claudius Heine
2019-03-25 8:13 ` Jan Kiszka
2019-03-25 8:21 ` Claudius Heine
2019-03-25 8:44 ` Jan Kiszka
2019-03-25 11:08 ` Jan Kiszka
2019-02-15 14:45 ` [PATCH v4 0/1] Template system Maxim Yu. Osipov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox