Re: Proposal for Adding OE/Yocto-Style Features Variables to Isar

["'Christopher Larson' via isar-users" <isar-users@googlegroups.com>]

On 9/26/2024 1:52 AM, Jan Kiszka wrote:
> On 25.09.24 22:27, 'Christopher Larson' via isar-users wrote:
>> Dear Isar Users,
>>
>> I would like to start a discussion about the possibility of supporting
>> OE/Yocto-style features variables within the Isar project. Currently,
>> Isar implements BASE_REPO_FEATURES and ROOTFS_FEATURES, which are quite
>> useful. However, I believe that adding support for DISTRO_FEATURES,
>> MACHINE_FEATURES, and possibly IMAGE_FEATURES would be worthwhile
>> additions to consider.
>>
>> I want to preface this by acknowledging that my perspective is
>> influenced by decades of experience with OpenEmbedded (OE) and OE-based
>> products. I recognize that Isar has a different philosophy, favoring
>> more direct approaches and fewer abstractions compared to OE.
>>
>> That said, I believe the value of these abstractions may justify the
>> added complexity. It seems that many downstreams end up reinventing
>> similar mechanisms for their own needs. For example, CIP adds
>> INSTALL_WIRELESS_TOOLS, USE_CIP_KERNEL_CONFIG, and CIP_IMAGE_OPTIONS,
>> the latter being a list of .inc files required by an image to allow for
>> metadata reuse. Our usage at Siemens includes similar reinventions as well.
>>
>> Certainly, we could leverage ROOTFS_FEATURES for certain rootfs/image
>> capabilities beyond the existing postprocessing in Isar. Establishing a
>> convention for including optional rootfs/image capabilities could avoid
>> metadata duplication, simplify managing development vs. production
>> filesystems, and provide customization mechanisms for downstreams.
>>
>> Regarding DISTRO_FEATURES and MACHINE_FEATURES, the Yocto documentation
>> covers them in general. The original intention was to allow for a
>> mechanism similar to Gentoo’s USE flags, coupled with OE’s three
>> orthogonal axes of distro, machine, and image. The intersection of these
>> would control the outcome, allowing any combination to be viable. This
>> results in machine support that is not tightly coupled to distro
>> capabilities or policy decisions, avoiding the pattern of each
>> downstream copying and modifying both distro and machine in a single
>> layer. This decoupling could prevent issues like machines installing
>> packages such as expand-on-first-boot unnecessarily.
>>
>> In OE, the intersection of these features determines certain
>> functionalities. A common example is hardware capabilities like WiFi or
>> Bluetooth, where the distro expresses a desire to support certain
>> functionalities. Only if both the distro and machine support it will the
>> required packages be installed.
>>
>> Details would need to be worked out, even if it is determined that this
>> provides more value than it adds in complexity. The core of the global
>> features in OE is their intersection in packagegroup-base, which
>> determines the default installed packages in images built from the
>> ground up. While this doesn’t make sense in Isar with a Debian base
>> image, there are still optional functionalities requiring package
>> installation. Often, this requires more than just a single
>> IMAGE_PREINSTALL line, so there’s value in having a simpler way to
>> express a desire to support that functionality. Isar may not need to
>> utilize this functionality directly, but it could be beneficial to
>> provide it for downstream use.
>>
>> Downstreams can and do implement functionality like this if they want
>> to, so I understand the argument for continuing this approach. However,
>> I believe there is value in providing basic functions to utilize such
>> capabilities and documented conventions for doing so consistently.
>>
>> I would love to hear what both Isar core developers and downstream
>> developers think about the possibility of providing a mechanism for
>> using variables like these. I believe that the ability to provide an
>> easier customization mechanism and an abstraction to better separate
>> concerns between the distro, machine, and images would be valuable. It
>> would also ease rootfs customization based on desired system features
>> (distro) and hardware capabilities (machine), if one uses these to
>> adjust ROOTFS_FEATURES.
>>
>> I don’t believe the default behavior of OE’s IMAGE_FEATURES, where
>> package lists are defined in FEATURE_PACKAGES_, is worth including here.
>> It’s not difficult for developers to manually implement package grouping
>> using features if needed, and it’s often better to create separate
>> packages if multiple dependencies should be pulled in at once.
>>
> 
> Thanks for the suggestion. As you mentioned already that you see
> potential to re-model and unify existing public layers with such a
> proposal, how about laying out how those would look like? Would make
> this discussion a bit more concrete, specifically for those of us not so
> long into OE like you are.

Greetings,

Thanks for all the replies to this, I appreciate the input. I'll start 
by considering OE/Yocto's usage of features, options for how to 
emphasize their use in isar, my personal recommendation for how to 
proceed, and finally look into what the usage would look like in a 
specific downstream example as suggested

To update the status on this, I've been working on a more concrete 
proposal to follow-up on the email discussion, as was requested, but 
it's not entirely clear as to the best approach. I feel like these are 
likely the main options:

     1. Do nothing, features can be implemented by downstream layers.
     2. Ease use of features by downstream layers, potentially by 
creating wrapper functions, or at the least default definitions of the 
features variables, and specifically COMBINED_FEATURES.
     3. Enforce more structure on features usage such as through 
inclusion of .inc or .bbclass files the way we have for features in our 
layer and cip has for image options.

OE/Yocto largely does the second, defining default values, and then 
using features in multiple places within the layer. They also provide a 
bbclass to require distro features for a given recipe, and an optional 
bbclass to ease use of features through overrides, and the 
packagegroup-base recipe which does not apply for us. Of course, we also 
don't intend to leverage the features within isar itself at this time 
the way oe-core does.

Wrapper functions tend to be avoided, in large part due to signature 
generation. BitBake's checksum generation for task signatures involves 
tracking variable dependencies and usage, and has special handling for 
bb.utils.contains(), which allows a variable to depend on the presence 
or absence of a feature from a features variable, without becoming 
dependent on the entirety of the features variable. Wrapping this 
function to ease calls would both be an added indirection that I feel 
adds an unnecessary abstraction to gain that slight ease of use, it will 
also prevent the signature generation from tracking feature usage fully.

I also feel that inclusion of a .inc or .bbclass for each feature is 
rather indirect, and doesn't align with how isar uses rootfs_features 
via a single bbclass extension for multiple features. I think this may 
be done by some downstreams, but is a tad too indirect for isar as a 
core layer.

To sum up, I feel that this proposal is largely to encourage the use of 
features variables and to add to the isar documentation to encourage 
their use, more than much direct implementation code. I'd think the 
following would be a good start:

     1. Add default DISTRO_FEATURES, MACHINE_FEATURES, COMBINED_FEATURES 
values, even if the first two are empty by default, and the latter 
should use inline python to ensure it's the intersection of the first two.
     2. Add to the docs to encourage use of the features variables, and 
specifically the use of bb.utils.contains, bb.utils.contains_any, and 
bb.utils.filter, to do so.
     3. Add a test using features to the test suite.
     4. Consider adding a class to ease use of features with overrides, 
as this can be helpful when defining many variables which are 
conditional on a feature. This should reduce the perceived need for 
feature-based file inclusion, I believe. distrooverrides.bbclass in 
oe-core or featureoverrides in sokol-flex could be candidate options to 
consider as a starting point.

As possible options for downstream layers to use features rather than 
custom solutions, we can consider possibilities for CIP_IMAGE_OPTIONS. 
Rather than isar-cip-core's CIP_IMAGE_OPTIONS listing .inc files to 
include, it could add an image extension bbclass which directly defines 
the variables based on ROOTFS_FEATURES or IMAGE_FEATURES, or it could 
keep the .inc inclusion and do so based on the features rather than a 
new variable. The main benefit there would be consistency rather than 
any other concrete benefit, however.

The benefit of the use of distro and machine I believe would aid in 
making the distro and machine less tightly bound, which should ease long 
term maintenance of the downstream layers, but this is difficult to show 
in a trivial example. I'll look into one of the public downstreams to 
try a PR as a suggested example.
-- 
Christopher Larson
Siemens AG
www.siemens.com

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/086d0928-13c2-478d-8e39-c89c15526912%40siemens.com.