Proposal for Adding OE/Yocto-Style Features Variables to Isar

Proposal for Adding OE/Yocto-Style Features Variables to Isar

['Christopher Larson' via isar-users]

Dear Isar Users,

I would like to start a discussion about the possibility of supporting 
OE/Yocto-style features variables within the Isar project. Currently, 
Isar implements BASE_REPO_FEATURES and ROOTFS_FEATURES, which are quite 
useful. However, I believe that adding support for DISTRO_FEATURES, 
MACHINE_FEATURES, and possibly IMAGE_FEATURES would be worthwhile 
additions to consider.

I want to preface this by acknowledging that my perspective is 
influenced by decades of experience with OpenEmbedded (OE) and OE-based 
products. I recognize that Isar has a different philosophy, favoring 
more direct approaches and fewer abstractions compared to OE.

That said, I believe the value of these abstractions may justify the 
added complexity. It seems that many downstreams end up reinventing 
similar mechanisms for their own needs. For example, CIP adds 
INSTALL_WIRELESS_TOOLS, USE_CIP_KERNEL_CONFIG, and CIP_IMAGE_OPTIONS, 
the latter being a list of .inc files required by an image to allow for 
metadata reuse. Our usage at Siemens includes similar reinventions as well.

Certainly, we could leverage ROOTFS_FEATURES for certain rootfs/image 
capabilities beyond the existing postprocessing in Isar. Establishing a 
convention for including optional rootfs/image capabilities could avoid 
metadata duplication, simplify managing development vs. production 
filesystems, and provide customization mechanisms for downstreams.

Regarding DISTRO_FEATURES and MACHINE_FEATURES, the Yocto documentation 
covers them in general. The original intention was to allow for a 
mechanism similar to Gentoo’s USE flags, coupled with OE’s three 
orthogonal axes of distro, machine, and image. The intersection of these 
would control the outcome, allowing any combination to be viable. This 
results in machine support that is not tightly coupled to distro 
capabilities or policy decisions, avoiding the pattern of each 
downstream copying and modifying both distro and machine in a single 
layer. This decoupling could prevent issues like machines installing 
packages such as expand-on-first-boot unnecessarily.

In OE, the intersection of these features determines certain 
functionalities. A common example is hardware capabilities like WiFi or 
Bluetooth, where the distro expresses a desire to support certain 
functionalities. Only if both the distro and machine support it will the 
required packages be installed.

Details would need to be worked out, even if it is determined that this 
provides more value than it adds in complexity. The core of the global 
features in OE is their intersection in packagegroup-base, which 
determines the default installed packages in images built from the 
ground up. While this doesn’t make sense in Isar with a Debian base 
image, there are still optional functionalities requiring package 
installation. Often, this requires more than just a single 
IMAGE_PREINSTALL line, so there’s value in having a simpler way to 
express a desire to support that functionality. Isar may not need to 
utilize this functionality directly, but it could be beneficial to 
provide it for downstream use.

Downstreams can and do implement functionality like this if they want 
to, so I understand the argument for continuing this approach. However, 
I believe there is value in providing basic functions to utilize such 
capabilities and documented conventions for doing so consistently.

I would love to hear what both Isar core developers and downstream 
developers think about the possibility of providing a mechanism for 
using variables like these. I believe that the ability to provide an 
easier customization mechanism and an abstraction to better separate 
concerns between the distro, machine, and images would be valuable. It 
would also ease rootfs customization based on desired system features 
(distro) and hardware capabilities (machine), if one uses these to 
adjust ROOTFS_FEATURES.

I don’t believe the default behavior of OE’s IMAGE_FEATURES, where 
package lists are defined in FEATURE_PACKAGES_, is worth including here. 
It’s not difficult for developers to manually implement package grouping 
using features if needed, and it’s often better to create separate 
packages if multiple dependencies should be pulled in at once.

Looking forward to your thoughts and feedback.

Best regards,
-- 
Christopher Larson
Siemens AG
www.siemens.com

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/27f2f225-d7c4-40dd-a4db-431eeb31cc52%40siemens.com.

Re: Proposal for Adding OE/Yocto-Style Features Variables to Isar

['Jan Kiszka' via isar-users]

On 25.09.24 22:27, 'Christopher Larson' via isar-users wrote:
> Dear Isar Users,
> 
> I would like to start a discussion about the possibility of supporting
> OE/Yocto-style features variables within the Isar project. Currently,
> Isar implements BASE_REPO_FEATURES and ROOTFS_FEATURES, which are quite
> useful. However, I believe that adding support for DISTRO_FEATURES,
> MACHINE_FEATURES, and possibly IMAGE_FEATURES would be worthwhile
> additions to consider.
> 
> I want to preface this by acknowledging that my perspective is
> influenced by decades of experience with OpenEmbedded (OE) and OE-based
> products. I recognize that Isar has a different philosophy, favoring
> more direct approaches and fewer abstractions compared to OE.
> 
> That said, I believe the value of these abstractions may justify the
> added complexity. It seems that many downstreams end up reinventing
> similar mechanisms for their own needs. For example, CIP adds
> INSTALL_WIRELESS_TOOLS, USE_CIP_KERNEL_CONFIG, and CIP_IMAGE_OPTIONS,
> the latter being a list of .inc files required by an image to allow for
> metadata reuse. Our usage at Siemens includes similar reinventions as well.
> 
> Certainly, we could leverage ROOTFS_FEATURES for certain rootfs/image
> capabilities beyond the existing postprocessing in Isar. Establishing a
> convention for including optional rootfs/image capabilities could avoid
> metadata duplication, simplify managing development vs. production
> filesystems, and provide customization mechanisms for downstreams.
> 
> Regarding DISTRO_FEATURES and MACHINE_FEATURES, the Yocto documentation
> covers them in general. The original intention was to allow for a
> mechanism similar to Gentoo’s USE flags, coupled with OE’s three
> orthogonal axes of distro, machine, and image. The intersection of these
> would control the outcome, allowing any combination to be viable. This
> results in machine support that is not tightly coupled to distro
> capabilities or policy decisions, avoiding the pattern of each
> downstream copying and modifying both distro and machine in a single
> layer. This decoupling could prevent issues like machines installing
> packages such as expand-on-first-boot unnecessarily.
> 
> In OE, the intersection of these features determines certain
> functionalities. A common example is hardware capabilities like WiFi or
> Bluetooth, where the distro expresses a desire to support certain
> functionalities. Only if both the distro and machine support it will the
> required packages be installed.
> 
> Details would need to be worked out, even if it is determined that this
> provides more value than it adds in complexity. The core of the global
> features in OE is their intersection in packagegroup-base, which
> determines the default installed packages in images built from the
> ground up. While this doesn’t make sense in Isar with a Debian base
> image, there are still optional functionalities requiring package
> installation. Often, this requires more than just a single
> IMAGE_PREINSTALL line, so there’s value in having a simpler way to
> express a desire to support that functionality. Isar may not need to
> utilize this functionality directly, but it could be beneficial to
> provide it for downstream use.
> 
> Downstreams can and do implement functionality like this if they want
> to, so I understand the argument for continuing this approach. However,
> I believe there is value in providing basic functions to utilize such
> capabilities and documented conventions for doing so consistently.
> 
> I would love to hear what both Isar core developers and downstream
> developers think about the possibility of providing a mechanism for
> using variables like these. I believe that the ability to provide an
> easier customization mechanism and an abstraction to better separate
> concerns between the distro, machine, and images would be valuable. It
> would also ease rootfs customization based on desired system features
> (distro) and hardware capabilities (machine), if one uses these to
> adjust ROOTFS_FEATURES.
> 
> I don’t believe the default behavior of OE’s IMAGE_FEATURES, where
> package lists are defined in FEATURE_PACKAGES_, is worth including here.
> It’s not difficult for developers to manually implement package grouping
> using features if needed, and it’s often better to create separate
> packages if multiple dependencies should be pulled in at once.
> 
> Looking forward to your thoughts and feedback.
> 
> Best regards,

Thanks for the suggestion. As you mentioned already that you see
potential to re-model and unify existing public layers with such a
proposal, how about laying out how those would look like? Would make
this discussion a bit more concrete, specifically for those of us not so
long into OE like you are.

Jan

-- 
Siemens AG, Technology
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/ce2b37a9-c0bb-41e3-bb81-40625472ee84%40siemens.com.

Re: Proposal for Adding OE/Yocto-Style Features Variables to Isar

['Christopher Larson' via isar-users]

On 9/26/2024 1:52 AM, Jan Kiszka wrote:
> On 25.09.24 22:27, 'Christopher Larson' via isar-users wrote:
>> Dear Isar Users,
>>
>> I would like to start a discussion about the possibility of supporting
>> OE/Yocto-style features variables within the Isar project. Currently,
>> Isar implements BASE_REPO_FEATURES and ROOTFS_FEATURES, which are quite
>> useful. However, I believe that adding support for DISTRO_FEATURES,
>> MACHINE_FEATURES, and possibly IMAGE_FEATURES would be worthwhile
>> additions to consider.
>>
>> I want to preface this by acknowledging that my perspective is
>> influenced by decades of experience with OpenEmbedded (OE) and OE-based
>> products. I recognize that Isar has a different philosophy, favoring
>> more direct approaches and fewer abstractions compared to OE.
>>
>> That said, I believe the value of these abstractions may justify the
>> added complexity. It seems that many downstreams end up reinventing
>> similar mechanisms for their own needs. For example, CIP adds
>> INSTALL_WIRELESS_TOOLS, USE_CIP_KERNEL_CONFIG, and CIP_IMAGE_OPTIONS,
>> the latter being a list of .inc files required by an image to allow for
>> metadata reuse. Our usage at Siemens includes similar reinventions as well.
>>
>> Certainly, we could leverage ROOTFS_FEATURES for certain rootfs/image
>> capabilities beyond the existing postprocessing in Isar. Establishing a
>> convention for including optional rootfs/image capabilities could avoid
>> metadata duplication, simplify managing development vs. production
>> filesystems, and provide customization mechanisms for downstreams.
>>
>> Regarding DISTRO_FEATURES and MACHINE_FEATURES, the Yocto documentation
>> covers them in general. The original intention was to allow for a
>> mechanism similar to Gentoo’s USE flags, coupled with OE’s three
>> orthogonal axes of distro, machine, and image. The intersection of these
>> would control the outcome, allowing any combination to be viable. This
>> results in machine support that is not tightly coupled to distro
>> capabilities or policy decisions, avoiding the pattern of each
>> downstream copying and modifying both distro and machine in a single
>> layer. This decoupling could prevent issues like machines installing
>> packages such as expand-on-first-boot unnecessarily.
>>
>> In OE, the intersection of these features determines certain
>> functionalities. A common example is hardware capabilities like WiFi or
>> Bluetooth, where the distro expresses a desire to support certain
>> functionalities. Only if both the distro and machine support it will the
>> required packages be installed.
>>
>> Details would need to be worked out, even if it is determined that this
>> provides more value than it adds in complexity. The core of the global
>> features in OE is their intersection in packagegroup-base, which
>> determines the default installed packages in images built from the
>> ground up. While this doesn’t make sense in Isar with a Debian base
>> image, there are still optional functionalities requiring package
>> installation. Often, this requires more than just a single
>> IMAGE_PREINSTALL line, so there’s value in having a simpler way to
>> express a desire to support that functionality. Isar may not need to
>> utilize this functionality directly, but it could be beneficial to
>> provide it for downstream use.
>>
>> Downstreams can and do implement functionality like this if they want
>> to, so I understand the argument for continuing this approach. However,
>> I believe there is value in providing basic functions to utilize such
>> capabilities and documented conventions for doing so consistently.
>>
>> I would love to hear what both Isar core developers and downstream
>> developers think about the possibility of providing a mechanism for
>> using variables like these. I believe that the ability to provide an
>> easier customization mechanism and an abstraction to better separate
>> concerns between the distro, machine, and images would be valuable. It
>> would also ease rootfs customization based on desired system features
>> (distro) and hardware capabilities (machine), if one uses these to
>> adjust ROOTFS_FEATURES.
>>
>> I don’t believe the default behavior of OE’s IMAGE_FEATURES, where
>> package lists are defined in FEATURE_PACKAGES_, is worth including here.
>> It’s not difficult for developers to manually implement package grouping
>> using features if needed, and it’s often better to create separate
>> packages if multiple dependencies should be pulled in at once.
>>
> 
> Thanks for the suggestion. As you mentioned already that you see
> potential to re-model and unify existing public layers with such a
> proposal, how about laying out how those would look like? Would make
> this discussion a bit more concrete, specifically for those of us not so
> long into OE like you are.

Greetings,

Thanks for all the replies to this, I appreciate the input. I'll start 
by considering OE/Yocto's usage of features, options for how to 
emphasize their use in isar, my personal recommendation for how to 
proceed, and finally look into what the usage would look like in a 
specific downstream example as suggested

To update the status on this, I've been working on a more concrete 
proposal to follow-up on the email discussion, as was requested, but 
it's not entirely clear as to the best approach. I feel like these are 
likely the main options:

     1. Do nothing, features can be implemented by downstream layers.
     2. Ease use of features by downstream layers, potentially by 
creating wrapper functions, or at the least default definitions of the 
features variables, and specifically COMBINED_FEATURES.
     3. Enforce more structure on features usage such as through 
inclusion of .inc or .bbclass files the way we have for features in our 
layer and cip has for image options.

OE/Yocto largely does the second, defining default values, and then 
using features in multiple places within the layer. They also provide a 
bbclass to require distro features for a given recipe, and an optional 
bbclass to ease use of features through overrides, and the 
packagegroup-base recipe which does not apply for us. Of course, we also 
don't intend to leverage the features within isar itself at this time 
the way oe-core does.

Wrapper functions tend to be avoided, in large part due to signature 
generation. BitBake's checksum generation for task signatures involves 
tracking variable dependencies and usage, and has special handling for 
bb.utils.contains(), which allows a variable to depend on the presence 
or absence of a feature from a features variable, without becoming 
dependent on the entirety of the features variable. Wrapping this 
function to ease calls would both be an added indirection that I feel 
adds an unnecessary abstraction to gain that slight ease of use, it will 
also prevent the signature generation from tracking feature usage fully.

I also feel that inclusion of a .inc or .bbclass for each feature is 
rather indirect, and doesn't align with how isar uses rootfs_features 
via a single bbclass extension for multiple features. I think this may 
be done by some downstreams, but is a tad too indirect for isar as a 
core layer.

To sum up, I feel that this proposal is largely to encourage the use of 
features variables and to add to the isar documentation to encourage 
their use, more than much direct implementation code. I'd think the 
following would be a good start:

     1. Add default DISTRO_FEATURES, MACHINE_FEATURES, COMBINED_FEATURES 
values, even if the first two are empty by default, and the latter 
should use inline python to ensure it's the intersection of the first two.
     2. Add to the docs to encourage use of the features variables, and 
specifically the use of bb.utils.contains, bb.utils.contains_any, and 
bb.utils.filter, to do so.
     3. Add a test using features to the test suite.
     4. Consider adding a class to ease use of features with overrides, 
as this can be helpful when defining many variables which are 
conditional on a feature. This should reduce the perceived need for 
feature-based file inclusion, I believe. distrooverrides.bbclass in 
oe-core or featureoverrides in sokol-flex could be candidate options to 
consider as a starting point.

As possible options for downstream layers to use features rather than 
custom solutions, we can consider possibilities for CIP_IMAGE_OPTIONS. 
Rather than isar-cip-core's CIP_IMAGE_OPTIONS listing .inc files to 
include, it could add an image extension bbclass which directly defines 
the variables based on ROOTFS_FEATURES or IMAGE_FEATURES, or it could 
keep the .inc inclusion and do so based on the features rather than a 
new variable. The main benefit there would be consistency rather than 
any other concrete benefit, however.

The benefit of the use of distro and machine I believe would aid in 
making the distro and machine less tightly bound, which should ease long 
term maintenance of the downstream layers, but this is difficult to show 
in a trivial example. I'll look into one of the public downstreams to 
try a PR as a suggested example.
-- 
Christopher Larson
Siemens AG
www.siemens.com

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/086d0928-13c2-478d-8e39-c89c15526912%40siemens.com.

Re: Proposal for Adding OE/Yocto-Style Features Variables to Isar

['Christopher Larson' via isar-users]

On 10/4/2024 10:20 AM, Christopher Larson wrote:
> On 9/26/2024 1:52 AM, Jan Kiszka wrote:
>> On 25.09.24 22:27, 'Christopher Larson' via isar-users wrote:
>>> I would like to start a discussion about the possibility of supporting
>>> OE/Yocto-style features variables within the Isar project. Currently,
>>> Isar implements BASE_REPO_FEATURES and ROOTFS_FEATURES, which are quite
>>> useful. However, I believe that adding support for DISTRO_FEATURES,
>>> MACHINE_FEATURES, and possibly IMAGE_FEATURES would be worthwhile
>>> additions to consider.
>>>
>>> I want to preface this by acknowledging that my perspective is
>>> influenced by decades of experience with OpenEmbedded (OE) and OE-based
>>> products. I recognize that Isar has a different philosophy, favoring
>>> more direct approaches and fewer abstractions compared to OE.
>>>
>>> That said, I believe the value of these abstractions may justify the
>>> added complexity. It seems that many downstreams end up reinventing
>>> similar mechanisms for their own needs. For example, CIP adds
>>> INSTALL_WIRELESS_TOOLS, USE_CIP_KERNEL_CONFIG, and CIP_IMAGE_OPTIONS,
>>> the latter being a list of .inc files required by an image to allow for
>>> metadata reuse. Our usage at Siemens includes similar reinventions as 
>>> well.
>>>
>>> Certainly, we could leverage ROOTFS_FEATURES for certain rootfs/image
>>> capabilities beyond the existing postprocessing in Isar. Establishing a
>>> convention for including optional rootfs/image capabilities could avoid
>>> metadata duplication, simplify managing development vs. production
>>> filesystems, and provide customization mechanisms for downstreams.
>>>
>>> Regarding DISTRO_FEATURES and MACHINE_FEATURES, the Yocto documentation
>>> covers them in general. The original intention was to allow for a
>>> mechanism similar to Gentoo’s USE flags, coupled with OE’s three
>>> orthogonal axes of distro, machine, and image. The intersection of these
>>> would control the outcome, allowing any combination to be viable. This
>>> results in machine support that is not tightly coupled to distro
>>> capabilities or policy decisions, avoiding the pattern of each
>>> downstream copying and modifying both distro and machine in a single
>>> layer. This decoupling could prevent issues like machines installing
>>> packages such as expand-on-first-boot unnecessarily.
>>>
>>> In OE, the intersection of these features determines certain
>>> functionalities. A common example is hardware capabilities like WiFi or
>>> Bluetooth, where the distro expresses a desire to support certain
>>> functionalities. Only if both the distro and machine support it will the
>>> required packages be installed.
>>>
>>> Details would need to be worked out, even if it is determined that this
>>> provides more value than it adds in complexity. The core of the global
>>> features in OE is their intersection in packagegroup-base, which
>>> determines the default installed packages in images built from the
>>> ground up. While this doesn’t make sense in Isar with a Debian base
>>> image, there are still optional functionalities requiring package
>>> installation. Often, this requires more than just a single
>>> IMAGE_PREINSTALL line, so there’s value in having a simpler way to
>>> express a desire to support that functionality. Isar may not need to
>>> utilize this functionality directly, but it could be beneficial to
>>> provide it for downstream use.
>>>
>>> Downstreams can and do implement functionality like this if they want
>>> to, so I understand the argument for continuing this approach. However,
>>> I believe there is value in providing basic functions to utilize such
>>> capabilities and documented conventions for doing so consistently.
>>>
>>> I would love to hear what both Isar core developers and downstream
>>> developers think about the possibility of providing a mechanism for
>>> using variables like these. I believe that the ability to provide an
>>> easier customization mechanism and an abstraction to better separate
>>> concerns between the distro, machine, and images would be valuable. It
>>> would also ease rootfs customization based on desired system features
>>> (distro) and hardware capabilities (machine), if one uses these to
>>> adjust ROOTFS_FEATURES.
>>>
>>> I don’t believe the default behavior of OE’s IMAGE_FEATURES, where
>>> package lists are defined in FEATURE_PACKAGES_, is worth including here.
>>> It’s not difficult for developers to manually implement package grouping
>>> using features if needed, and it’s often better to create separate
>>> packages if multiple dependencies should be pulled in at once.
>>>
>>
>> Thanks for the suggestion. As you mentioned already that you see
>> potential to re-model and unify existing public layers with such a
>> proposal, how about laying out how those would look like? Would make
>> this discussion a bit more concrete, specifically for those of us not so
>> long into OE like you are.
> 
> Thanks for all the replies to this, I appreciate the input. I'll start 
> by considering OE/Yocto's usage of features, options for how to 
> emphasize their use in isar, my personal recommendation for how to 
> proceed, and finally look into what the usage would look like in a 
> specific downstream example as suggested
> 
> To update the status on this, I've been working on a more concrete 
> proposal to follow-up on the email discussion, as was requested, but 
> it's not entirely clear as to the best approach. I feel like these are 
> likely the main options:
> 
>      1. Do nothing, features can be implemented by downstream layers.
>      2. Ease use of features by downstream layers, potentially by 
> creating wrapper functions, or at the least default definitions of the 
> features variables, and specifically COMBINED_FEATURES.
>      3. Enforce more structure on features usage such as through 
> inclusion of .inc or .bbclass files the way we have for features in our 
> layer and cip has for image options.
> 
> OE/Yocto largely does the second, defining default values, and then 
> using features in multiple places within the layer. They also provide a 
> bbclass to require distro features for a given recipe, and an optional 
> bbclass to ease use of features through overrides, and the packagegroup- 
> base recipe which does not apply for us. Of course, we also don't intend 
> to leverage the features within isar itself at this time the way oe-core 
> does.
> 
> Wrapper functions tend to be avoided, in large part due to signature 
> generation. BitBake's checksum generation for task signatures involves 
> tracking variable dependencies and usage, and has special handling for 
> bb.utils.contains(), which allows a variable to depend on the presence 
> or absence of a feature from a features variable, without becoming 
> dependent on the entirety of the features variable. Wrapping this 
> function to ease calls would both be an added indirection that I feel 
> adds an unnecessary abstraction to gain that slight ease of use, it will 
> also prevent the signature generation from tracking feature usage fully.
> 
> I also feel that inclusion of a .inc or .bbclass for each feature is 
> rather indirect, and doesn't align with how isar uses rootfs_features 
> via a single bbclass extension for multiple features. I think this may 
> be done by some downstreams, but is a tad too indirect for isar as a 
> core layer.
> 
> To sum up, I feel that this proposal is largely to encourage the use of 
> features variables and to add to the isar documentation to encourage 
> their use, more than much direct implementation code. I'd think the 
> following would be a good start:
> 
>      1. Add default DISTRO_FEATURES, MACHINE_FEATURES, COMBINED_FEATURES 
> values, even if the first two are empty by default, and the latter 
> should use inline python to ensure it's the intersection of the first two.
>      2. Add to the docs to encourage use of the features variables, and 
> specifically the use of bb.utils.contains, bb.utils.contains_any, and 
> bb.utils.filter, to do so.
>      3. Add a test using features to the test suite.
>      4. Consider adding a class to ease use of features with overrides, 
> as this can be helpful when defining many variables which are 
> conditional on a feature. This should reduce the perceived need for 
> feature-based file inclusion, I believe. distrooverrides.bbclass in oe- 
> core or featureoverrides in sokol-flex could be candidate options to 
> consider as a starting point.
> 
> As possible options for downstream layers to use features rather than 
> custom solutions, we can consider possibilities for CIP_IMAGE_OPTIONS. 
> Rather than isar-cip-core's CIP_IMAGE_OPTIONS listing .inc files to 
> include, it could add an image extension bbclass which directly defines 
> the variables based on ROOTFS_FEATURES or IMAGE_FEATURES, or it could 
> keep the .inc inclusion and do so based on the features rather than a 
> new variable. The main benefit there would be consistency rather than 
> any other concrete benefit, however.
> 
> The benefit of the use of distro and machine I believe would aid in 
> making the distro and machine less tightly bound, which should ease long 
> term maintenance of the downstream layers, but this is difficult to show 
> in a trivial example. I'll look into one of the public downstreams to 
> try a PR as a suggested example.

I've gone back over isar-cip-core and looked at its usage of 
CIP_IMAGE_OPTIONS. Currently the isar core images include any files 
listed in this variable, but it's only currently set by the swupdate and 
ebg-swu kas configurations, along with many other variables. Those kas 
configurations also add swupdate to OVERRIDES directly, which then 
alters the included feed configurations.

So this appears to be a case where we have a global change to 
configuration, and then multiple corresponding changes to image 
construction. These sets of configurations in the kas configs are not 
particularly accessible for those not using kas to build, and the 
CIP_IMAGE_OPTIONS contents (i.e. "recipes-core/images/swupdate.inc") are 
difficult to check conditionally for a downstream of isar-cip-core. The 
kas configs have a number of configurations set, as do the .inc files 
included by CIP_IMAGE_OPTIONS.

This, to me, is a good example of an implementation that would benefit 
from features, as they provide a way for the user to enable 
functionality whether via kas or otherwise. As pointed out by Cedric, 
the features list also functions as documentation, in that 
ROOTFS_FEATURES and IMAGE_FEATURES help describe the behavior of an 
image, and could potentially be included in a file in the image to help 
determine image behavior and contents after the fact. It also makes it 
easier for a downstream to alter the image further based on the enabling 
of the feature, as it's trivial to bbappend or add an image class that 
acts when a given feature is enabled. Admittedly, most downstreams would 
provide their own images, but this would provide a pattern they could 
follow as well, which would increase consistency between the layers.

To handle the global configuration change (feeds added, etc), either we 
could operate conditionally on IMAGE_FEATURES globally, but this assumes 
it's not allowed to be different on a per-image basis, which isn't 
typical, or one could add a DISTRO_FEATURES += "swupdate" and then 
conditionally operate on this elsewhere rather than using a custom 
override, and then potentially enable the image feature by default based 
on this distro feature.

This does add an indirection, but I think the value of having easy to 
set and check lists of features for 1. documentation, 2. ease of use for 
enabling/disabling, and 3. easy to act on conditionally in a downstream, 
makes it worth the indirection. This is one of the simpler proof of 
concept areas I could find that still shows the value.

It's not entirely clear to me which settings in the swupdate.yml and 
ebg-swu.yml are appropriate for always enabling based on the feature vs 
being provided as examples the user is expected to override, so I didn't 
move forward with the PR just yet.
-- 
Christopher Larson
Siemens AG
www.siemens.com

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/5ddf60dc-f297-4d5f-9de2-837fdd0e7e75%40siemens.com.

Re: Proposal for Adding OE/Yocto-Style Features Variables to Isar

[Baurzhan Ismagulov]

Hello Christopher,

On 2024-09-25 13:27, 'Christopher Larson' via isar-users wrote:
> I would like to start a discussion about the possibility of supporting
> OE/Yocto-style features variables within the Isar project. Currently, Isar
> implements BASE_REPO_FEATURES and ROOTFS_FEATURES, which are quite useful.
> However, I believe that adding support for DISTRO_FEATURES,
> MACHINE_FEATURES, and possibly IMAGE_FEATURES would be worthwhile additions
> to consider.

Anton had looked at this a couple of year ago; I'd suggest to hear his insights
on this. He should be back next week.

With kind regards,
Baurzhan

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/ZvUlm7t9eAMi6RTp%40abai.m.ilbers.de.

Re: Proposal for Adding OE/Yocto-Style Features Variables to Isar

[Anton Mikanovich]

25/09/2024 23:27, 'Christopher Larson' via isar-users wrote:
> Dear Isar Users,
>
> I would like to start a discussion about the possibility of supporting 
> OE/Yocto-style features variables within the Isar project. Currently, 
> Isar implements BASE_REPO_FEATURES and ROOTFS_FEATURES, which are 
> quite useful. However, I believe that adding support for 
> DISTRO_FEATURES, MACHINE_FEATURES, and possibly IMAGE_FEATURES would 
> be worthwhile additions to consider.
>
> I want to preface this by acknowledging that my perspective is 
> influenced by decades of experience with OpenEmbedded (OE) and 
> OE-based products. I recognize that Isar has a different philosophy, 
> favoring more direct approaches and fewer abstractions compared to OE.
>
> That said, I believe the value of these abstractions may justify the 
> added complexity. It seems that many downstreams end up reinventing 
> similar mechanisms for their own needs. For example, CIP adds 
> INSTALL_WIRELESS_TOOLS, USE_CIP_KERNEL_CONFIG, and CIP_IMAGE_OPTIONS, 
> the latter being a list of .inc files required by an image to allow 
> for metadata reuse. Our usage at Siemens includes similar reinventions 
> as well.
>
> Certainly, we could leverage ROOTFS_FEATURES for certain rootfs/image 
> capabilities beyond the existing postprocessing in Isar. Establishing 
> a convention for including optional rootfs/image capabilities could 
> avoid metadata duplication, simplify managing development vs. 
> production filesystems, and provide customization mechanisms for 
> downstreams.
>
> Regarding DISTRO_FEATURES and MACHINE_FEATURES, the Yocto 
> documentation covers them in general. The original intention was to 
> allow for a mechanism similar to Gentoo’s USE flags, coupled with OE’s 
> three orthogonal axes of distro, machine, and image. The intersection 
> of these would control the outcome, allowing any combination to be 
> viable. This results in machine support that is not tightly coupled to 
> distro capabilities or policy decisions, avoiding the pattern of each 
> downstream copying and modifying both distro and machine in a single 
> layer. This decoupling could prevent issues like machines installing 
> packages such as expand-on-first-boot unnecessarily.
>
> In OE, the intersection of these features determines certain 
> functionalities. A common example is hardware capabilities like WiFi 
> or Bluetooth, where the distro expresses a desire to support certain 
> functionalities. Only if both the distro and machine support it will 
> the required packages be installed.
>
> Details would need to be worked out, even if it is determined that 
> this provides more value than it adds in complexity. The core of the 
> global features in OE is their intersection in packagegroup-base, 
> which determines the default installed packages in images built from 
> the ground up. While this doesn’t make sense in Isar with a Debian 
> base image, there are still optional functionalities requiring package 
> installation. Often, this requires more than just a single 
> IMAGE_PREINSTALL line, so there’s value in having a simpler way to 
> express a desire to support that functionality. Isar may not need to 
> utilize this functionality directly, but it could be beneficial to 
> provide it for downstream use.
>
> Downstreams can and do implement functionality like this if they want 
> to, so I understand the argument for continuing this approach. 
> However, I believe there is value in providing basic functions to 
> utilize such capabilities and documented conventions for doing so 
> consistently.
>
> I would love to hear what both Isar core developers and downstream 
> developers think about the possibility of providing a mechanism for 
> using variables like these. I believe that the ability to provide an 
> easier customization mechanism and an abstraction to better separate 
> concerns between the distro, machine, and images would be valuable. It 
> would also ease rootfs customization based on desired system features 
> (distro) and hardware capabilities (machine), if one uses these to 
> adjust ROOTFS_FEATURES.
>
> I don’t believe the default behavior of OE’s IMAGE_FEATURES, where 
> package lists are defined in FEATURE_PACKAGES_, is worth including 
> here. It’s not difficult for developers to manually implement package 
> grouping using features if needed, and it’s often better to create 
> separate packages if multiple dependencies should be pulled in at once.
>
> Looking forward to your thoughts and feedback.
>
> Best regards,

Hello Christopher,

I agree that adding at least some part of OE-style 'features' can be useful
for existing and potential downstreams, especially for big vendor layers.
But we should carefully think about the implementation itself, because, as
you've already mentioned, Isar can only provide an APIs (without adding too
much packages into base minimal image), but every API should be used in
meta-isar/meta-test layers and covered by CI.
Maybe we can migrate our current example packages from local.conf to some
new package group with adding 'features' filtering as a first step.

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/0ff24b5e-41a7-4aab-8a02-906774cf8af8%40ilbers.de.