[RFC PATCH 0/2] Add bbclass to generate build configuration from snippets

[RFC PATCH 0/2] Add bbclass to generate build configuration from snippets

[Q. Gylstorff]

From: Quirin Gylstorff <quirin.gylstorff@siemens.com>

This patchset adds the `build-config-snippets.bbclass`. This class can be used
to generate package dependencies according to the used build configuration.

It is in use as 'kconfig-snippets.bbclass' in [1] for the swupdate configuration.


[1]: https://gitlab.com/cip-project/cip-core/isar-cip-core/

Quirin Gylstorff (2):
  Add build-config-snippet class
  Add example for build-config-snippets

 doc/technical_overview.md                     | 33 +++++++
 meta-isar/conf/local.conf.sample              |  2 +-
 .../example-build-config_0.1.bb               | 32 +++++++
 .../example-build-config/files/build.snippet  |  1 +
 .../example-build-config/files/build_config   |  1 +
 meta/classes/build-config-snippets.bbclass    | 87 +++++++++++++++++++
 6 files changed, 155 insertions(+), 1 deletion(-)
 create mode 100644 meta-isar/recipes-app/example-build-config/example-build-config_0.1.bb
 create mode 100644 meta-isar/recipes-app/example-build-config/files/build.snippet
 create mode 100644 meta-isar/recipes-app/example-build-config/files/build_config
 create mode 100644 meta/classes/build-config-snippets.bbclass

-- 
2.20.1

[RFC PATCH 1/2] Add build-config-snippet class

[Q. Gylstorff]

From: Quirin Gylstorff <quirin.gylstorff@siemens.com>

This class allows to generate a build configuration with debian
dependencies.

Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
---
 doc/technical_overview.md                  | 33 ++++++++
 meta/classes/build-config-snippets.bbclass | 87 ++++++++++++++++++++++
 2 files changed, 120 insertions(+)
 create mode 100644 meta/classes/build-config-snippets.bbclass

diff --git a/doc/technical_overview.md b/doc/technical_overview.md
index bbd22b6..fdfec0c 100644
--- a/doc/technical_overview.md
+++ b/doc/technical_overview.md
@@ -308,3 +308,36 @@ to the end.
 Only template files from the `WORKDIR` are accepted. Either specify relative
 paths based on the recipes `WORKDIR` or absolute paths containing the `WORKDIR`
 in the `TEMPLATE_FILES` variable.
+
+### 3.9.2 Build configuration snippets
+
+A build configuration can be created from multiple snippets files.
+
+This system is implemented in the `build-config-snippets.bbclass` and defines
+a `do_generate_build_configuration` task.
+
+To define a feature set, the user has to define the following
+variable to an empty string:
+
+```
+BUILD_FEATURE_featurename = ""
+```
+
+Then, required additions to the variables can be defined:
+
+```
+BUILD_FEATURE_featurename[BUILD_CONFIG_SNIPPETS] = "file://snippet-file-name.snippet"
+BUILD_FEATURE_featurename[SRC_URI] = "file://required-file.txt"
+BUILD_FEATURE_featurename[DEPENDS] = "deb-pkg1 deb-pkg2 deb-pkg3"
+BUILD_FEATURE_featurename[DEBIAN_DEPENDS] = "deb-pkg1"
+BUILD_FEATURE_featurename[DEBIAN_BUILD_DEPENDS] = "deb-pkg1,deb-pkg2,deb-pkg3"
+```
+
+The `BUILD_CONFIG_SNIPPETS` flag gives a list of URI entries, where only
+file:// is supported. These snippets are appended to the BUILD_CONFIG file.
+
+Features can depend on other features via the following mechanism:
+
+```
+BUILD_FEATURE_DEPS[feature1] = "feature2"
+```
diff --git a/meta/classes/build-config-snippets.bbclass b/meta/classes/build-config-snippets.bbclass
new file mode 100644
index 0000000..1d1054d
--- /dev/null
+++ b/meta/classes/build-config-snippets.bbclass
@@ -0,0 +1,87 @@
+# This software is a part of ISAR.
+# Copyright (C) 2021 Siemens AG
+#
+# SPDX-License-Identifier: MIT
+
+
+BUILD_CONFIG_SNIPPETS = ""
+
+# The following function defines the build config snippet system
+# with automatich debian dependency injection
+#
+# To define a feature set, the user has to define the following
+# variable to an empty string:
+#
+# BUILD_FEATURE_featurename = ""
+#
+# Then, required additions to the variables can be defined:
+#
+# BUILD_FEATURE_featurename[BUILD_CONFIG_SNIPPETS] = "file://snippet-file-name.snippet"
+# BUILD_FEATURE_featurename[SRC_URI] = "file://required-file.txt"
+# BUILD_FEATURE_featurename[DEPENDS] = "deb-pkg1 deb-pkg2 deb-pkg3"
+# BUILD_FEATURE_featurename[DEBIAN_DEPENDS] = "deb-pkg1"
+# BUILD_FEATURE_featurename[DEBIAN_BUILD_DEPENDS] = "deb-pkg1,deb-pkg2,deb-pkg3"
+
+# The 'BUILD_CONFIG_SNIPPETS' flag gives a list of URI entries, where only
+# file:// is supported. These snippets are appended to the BUILD_CONFIG file.
+#
+# Features can depend on other features via the following mechanism:
+#
+# BUILD_FEATURE_DEPS[feature1] = "feature2"
+
+KOMMA_SEPERATED_ELEMENTS ?= "DEBIAN_BUILD_DEPENDS DEBIAN_DEPENDS"
+
+python () {
+    requested_features = d.getVar("BUILD_FEATURES", True) or ""
+
+    features = set(requested_features.split())
+    old_features = set()
+    feature_deps = d.getVarFlags("BUILD_FEATURE_DEPS") or {}
+    while old_features != features:
+        diff_features = old_features.symmetric_difference(features)
+        old_features = features.copy()
+        for i in diff_features:
+            features.update(feature_deps.get(i, "").split())
+
+    for f in sorted(features):
+        bb.debug(2, "Feature: " + f)
+        varname = "BUILD_FEATURE_" + f
+        dummyvar = d.getVar(varname, False)
+        if dummyvar == None:
+            bb.error("Feature var " + f + " must be defined with needed flags.")
+        else:
+            feature_flags = d.getVarFlags(varname)
+            for feature_varname in sorted(feature_flags):
+                if feature_flags.get(feature_varname, "") != "":
+                    sep = " "
+
+                    # Required to add BUILD_CONFIG_SNIPPETS to SRC_URI here,
+                    # because 'SRC_URI += "${BUILD_CONFIG_SNIPPETS}"' would
+                    # conflict with SRC_APT feature.
+                    if feature_varname == "BUILD_CONFIG_SNIPPETS":
+                        d.appendVar('SRC_URI',
+                            " " + feature_flags[feature_varname].strip())
+
+                    # BUILD_DEP_DEPENDS and DEBIAN_DEPENDS is ',' separated
+                    # Only add ',' if there is already something there
+                    if feature_varname in (d.getVar("KOMMA_SEPERATED_ELEMENTS", True) or "").split():
+                        sep = "," if d.getVar(feature_varname) else ""
+
+                    d.appendVar(feature_varname,
+                        sep + feature_flags[feature_varname].strip())
+}
+
+# BUILD_CONFIG must be a predefined bitbake variable and the corresponding file
+# must exist in the WORKDIR.
+# The resulting generated config is the same file suffixed with ".gen"
+
+do_generate_build_configuration() {
+        GENCONFIG="${WORKDIR}/${BUILD_CONFIG}".gen
+        rm -f "$GENCONFIG"
+        cp "${WORKDIR}/${BUILD_CONFIG}" "$GENCONFIG"
+        for CONFIG_SNIPPET in $(echo "${BUILD_CONFIG_SNIPPETS}" | sed 's#file://##g')
+        do
+                cat ${WORKDIR}/$CONFIG_SNIPPET >> "$GENCONFIG"
+        done
+}
+addtask generate_build_configuration before do_prepare_build after do_patch
-- 
2.20.1

[RFC PATCH 2/2] Add example for build-config-snippets

[Q. Gylstorff]

From: Quirin Gylstorff <quirin.gylstorff@siemens.com>

Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
---
 meta-isar/conf/local.conf.sample              |  2 +-
 .../example-build-config_0.1.bb               | 32 +++++++++++++++++++
 .../example-build-config/files/build.snippet  |  1 +
 .../example-build-config/files/build_config   |  1 +
 4 files changed, 35 insertions(+), 1 deletion(-)
 create mode 100644 meta-isar/recipes-app/example-build-config/example-build-config_0.1.bb
 create mode 100644 meta-isar/recipes-app/example-build-config/files/build.snippet
 create mode 100644 meta-isar/recipes-app/example-build-config/files/build_config

diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 6cf1656..b86d618 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -178,7 +178,7 @@ CONF_VERSION = "1"
 
 #
 # The default list of extra packages to be installed.
-IMAGE_INSTALL = "hello-isar example-raw example-module-${KERNEL_NAME} enable-fsck isar-exclude-docs samefile hello isar-disable-apt-cache cowsay example-prebuilt"
+IMAGE_INSTALL = "hello-isar example-raw example-module-${KERNEL_NAME} enable-fsck isar-exclude-docs samefile hello isar-disable-apt-cache cowsay example-prebuilt example-build-config"
 
 #
 # Enable cross-compilation support
diff --git a/meta-isar/recipes-app/example-build-config/example-build-config_0.1.bb b/meta-isar/recipes-app/example-build-config/example-build-config_0.1.bb
new file mode 100644
index 0000000..c899aed
--- /dev/null
+++ b/meta-isar/recipes-app/example-build-config/example-build-config_0.1.bb
@@ -0,0 +1,32 @@
+# Sample application using dpkg-raw, which turns a folder (${D}) of
+# files into a .deb
+#
+# This software is a part of ISAR.
+
+DESCRIPTION = "Sample use for build-config-snippets for ISAR"
+MAINTAINER = "Your name here <you@domain.com>"
+
+inherit dpkg-raw
+inherit build-config-snippets
+BUILD_CONFIG ?= "build_config"
+
+SRC_URI += "file://${BUILD_CONFIG}"
+
+BUILD_FEATURE_libhello = ""
+BUILD_FEATURE_libhello[DEPENDS] = "libhello"
+BUILD_FEATURE_libhello[DEBIAN_DEPENDS] = "libhello"
+BUILD_FEATURE_libhello[DEBIAN_BUILD_DEPENDS] = "libhello"
+BUILD_FEATURE_libhello[BUILD_CONFIG_SNIPPETS] = "file://build.snippet"
+
+
+BUILD_FEATURES += "libhello"
+do_prepare_build[cleandirs] = "${D}/usr/share/example-build-config"
+do_prepare_build() {
+    # use deb_debianize to generate the necessary debian files
+    deb_debianize
+
+    # ${BUILD_CONFIG}.gen contains the concated build_config
+    install ${WORKDIR}/build_config.gen ${D}/usr/share/example-build-config/build_config.gen
+}
+
+
diff --git a/meta-isar/recipes-app/example-build-config/files/build.snippet b/meta-isar/recipes-app/example-build-config/files/build.snippet
new file mode 100644
index 0000000..199e48a
--- /dev/null
+++ b/meta-isar/recipes-app/example-build-config/files/build.snippet
@@ -0,0 +1 @@
+# build snippet test
diff --git a/meta-isar/recipes-app/example-build-config/files/build_config b/meta-isar/recipes-app/example-build-config/files/build_config
new file mode 100644
index 0000000..d95da26
--- /dev/null
+++ b/meta-isar/recipes-app/example-build-config/files/build_config
@@ -0,0 +1 @@
+# BUILD CONFIG EXAMPLE
-- 
2.20.1

Re: [RFC PATCH 0/2] Add bbclass to generate build configuration from snippets

[Jan Kiszka]

On 09.08.21 12:26, Q. Gylstorff wrote:
> From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
> 
> This patchset adds the `build-config-snippets.bbclass`. This class can be used
> to generate package dependencies according to the used build configuration.
> 
> It is in use as 'kconfig-snippets.bbclass' in [1] for the swupdate configuration.
> 

Yeah, but working around the anti-distro properties of that packages. A
proper SWUpdate would build against all features and be configurable
during runtime.

Given that this is an anti pattern, I'm still not sure about what to do
with it. We are carrying it for SWUpdate for quite a while but that's
still the only user. Do we really see others? Or should we better invest
in fixing SWUpdate so that we can use its original Debian package, one day?

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [RFC PATCH 0/2] Add bbclass to generate build configuration from snippets

[Gylstorff Quirin]

On 8/9/21 2:58 PM, Jan Kiszka wrote:
> On 09.08.21 12:26, Q. Gylstorff wrote:
>> From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>
>> This patchset adds the `build-config-snippets.bbclass`. This class can be used
>> to generate package dependencies according to the used build configuration.
>>
>> It is in use as 'kconfig-snippets.bbclass' in [1] for the swupdate configuration.
>>
> 
> Yeah, but working around the anti-distro properties of that packages. A
> proper SWUpdate would build against all features and be configurable
> during runtime.
> 

We have the bootloader feature which is exclusive selected. So u-boot or 
grub or efibootguard.

> Given that this is an anti pattern, I'm still not sure about what to do
> with it. We are carrying it for SWUpdate for quite a while but that's
> still the only user. Do we really see others? Or should we better invest
> in fixing SWUpdate so that we can use its original Debian package, one day?
> 

I don't see others at the moment. The alternative would be make one 
config for everything and only add the bootloader as configuration snippet.


> Jan
>

Quirin

Re: [RFC PATCH 0/2] Add bbclass to generate build configuration from snippets

[Gylstorff Quirin]

On 8/9/21 3:46 PM, Gylstorff Quirin wrote:
> 
> 
> On 8/9/21 2:58 PM, Jan Kiszka wrote:
>> On 09.08.21 12:26, Q. Gylstorff wrote:
>>> From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>>
>>> This patchset adds the `build-config-snippets.bbclass`. This class 
>>> can be used
>>> to generate package dependencies according to the used build 
>>> configuration.
>>>
>>> It is in use as 'kconfig-snippets.bbclass' in [1] for the swupdate 
>>> configuration.
>>>
>>
>> Yeah, but working around the anti-distro properties of that packages. A
>> proper SWUpdate would build against all features and be configurable
>> during runtime.
>>
> 
> We have the bootloader feature which is exclusive selected. So u-boot or 
> grub or efibootguard.
> 

I forgot the embedded lua code.

>> Given that this is an anti pattern, I'm still not sure about what to do
>> with it. We are carrying it for SWUpdate for quite a while but that's
>> still the only user. Do we really see others? Or should we better invest
>> in fixing SWUpdate so that we can use its original Debian package, one 
>> day?
>>
> 
> I don't see others at the moment. The alternative would be make one 
> config for everything and only add the bootloader as configuration snippet.
> 
> 
>> Jan
>>
> 
> Quirin
>
Quirin