'IMAGE_FEATURES' in Isar

'IMAGE_FEATURES' in Isar

[Claudius Heine]

Hi,

since we discussed that we might want to implement 'IMAGE_FEATURES' in 
Isar in order to support images without kernel, I wanted to start some 
brainstorming about what this should be used in other cases.

While discussing that with Jan we came some ideas that would be nice to 
be implemented as features:

   - Cleanup steps like localepurge, and apt-get clean
   - Removal of apt and other possible reduction tasks
   - ??? other ideas?

About installing the kernel: It could be possible to have some 
'nokernel' feature, but I am currently not sure if that is right.
Debians 'debootstrap' does not install a kernel per default, so if we 
now have a kernel per opt-out we are a bit in a conflict with the 
'Debian-Way-Of-Life' IMO. Because there even the kernel is a opt-in 
package. Of course most images created by Isar will have a kernel, so I 
also see that reason for including one.

The other question might be if having a kernel should be implemented as 
a IMAGE_FEATURE or rather a setting in the machine.conf? In the simplest 
solution maybe something like:

   machine.conf:
   KERNEL='yes'

while the multiconfig specifies which kernel to use and the distro what 
kernels are available, like it does today.

What do you think?

Thanks,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: 'IMAGE_FEATURES' in Isar

[Baurzhan Ismagulov]

On Wed, Apr 11, 2018 at 02:59:38PM +0200, Claudius Heine wrote:
> since we discussed that we might want to implement 'IMAGE_FEATURES' in Isar
> in order to support images without kernel, I wanted to start some
> brainstorming about what this should be used in other cases.

Why not just generate without the kernel by default, adding one (possibly via a
virtualized name) to specific images as a dependency / IMAGE_INSTALL? IMHO,
the logic of adding instead of removing is more in the spirit of bitbake.


> While discussing that with Jan we came some ideas that would be nice to be
> implemented as features:
> 
>   - Cleanup steps like localepurge, and apt-get clean

Just do it always, as in your patches?


>   - Removal of apt and other possible reduction tasks

Additional recipe some images depend on?


> About installing the kernel: It could be possible to have some 'nokernel'
> feature, but I am currently not sure if that is right.
> Debians 'debootstrap' does not install a kernel per default, so if we now
> have a kernel per opt-out we are a bit in a conflict with the
> 'Debian-Way-Of-Life' IMO. Because there even the kernel is a opt-in package.

I agree. Specify explicitly where needed, not remove some enforced default.
There are systems where the kernel is installed in a separate partition and
isn't contained in the rootfs.


> Of course most images created by Isar will have a kernel, so I also see that
> reason for including one.

Not a problem with explicit addition.


> The other question might be if having a kernel should be implemented as a
> IMAGE_FEATURE or rather a setting in the machine.conf?

We already have it in IMAGE_INSTALL for isar-image-base. Doesn't it work if the
kernel is removed from there?


With kind regards,
Baurzhan.

Re: 'IMAGE_FEATURES' in Isar

[Claudius Heine]

Hi Baurzhan,

On 2018-04-12 16:26, Baurzhan Ismagulov wrote:
> On Wed, Apr 11, 2018 at 02:59:38PM +0200, Claudius Heine wrote:
>> since we discussed that we might want to implement 'IMAGE_FEATURES' in Isar
>> in order to support images without kernel, I wanted to start some
>> brainstorming about what this should be used in other cases.
> 
> Why not just generate without the kernel by default, adding one (possibly via a
> virtualized name) to specific images as a dependency / IMAGE_INSTALL? IMHO,
> the logic of adding instead of removing is more in the spirit of bitbake.

I just posted an idea to solve this in a simple way, that shows how I 
would have done it.

Alex brought up the IMAGE_FEATURES idea and this thread just discuss it 
and to answer the questions: What else could IMAGE_FEATURES be used for? 
And does it make sense to use it to specify if the kernel should be part 
of the image or not?

>> While discussing that with Jan we came some ideas that would be nice to be
>> implemented as features:
>>
>>    - Cleanup steps like localepurge, and apt-get clean
> 
> Just do it always, as in your patches?

You meant the '*-configscripts.sh'?

AFAIK the purpose is to try to move most of this stuff out of there and 
into a 'system-setup-package' of sorts. At lease for all of those 
things, that don't require doing it at the end of everything.

The rest of those steps (like those cleanup commands) should be split up 
into smaller scripts each with just one task and then trigger them via 
IMAGE_FEATURES after all packages were installed. Then the difference 
between raspbian and debian might just be some different entries in the 
IMAGE_FEATURES (or DISTRO_FEATURES) variable. At least thats my current 
idea here.

>>    - Removal of apt and other possible reduction tasks
> 
> Additional recipe some images depend on?

Removal of apt and other reduction steps should be done after all 
packages are installed using apt. Doing that in some other recipe/debian 
package just doesn't work.

>> About installing the kernel: It could be possible to have some 'nokernel'
>> feature, but I am currently not sure if that is right.
>> Debians 'debootstrap' does not install a kernel per default, so if we now
>> have a kernel per opt-out we are a bit in a conflict with the
>> 'Debian-Way-Of-Life' IMO. Because there even the kernel is a opt-in package.
> 
> I agree. Specify explicitly where needed, not remove some enforced default.
> There are systems where the kernel is installed in a separate partition and
> isn't contained in the rootfs.
> 
> 
>> Of course most images created by Isar will have a kernel, so I also see that
>> reason for including one.
> 
> Not a problem with explicit addition.
> 
> 
>> The other question might be if having a kernel should be implemented as a
>> IMAGE_FEATURE or rather a setting in the machine.conf?
> 
> We already have it in IMAGE_INSTALL for isar-image-base. Doesn't it work if the
> kernel is removed from there?

This is how I do it in the current "Optional kernel" patchset.

regards,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: 'IMAGE_FEATURES' in Isar

[Baurzhan Ismagulov]

On Thu, Apr 12, 2018 at 05:35:17PM +0200, Claudius Heine wrote:
> Alex brought up the IMAGE_FEATURES idea and this thread just discuss it and
> to answer the questions: What else could IMAGE_FEATURES be used for? And
> does it make sense to use it to specify if the kernel should be part of the
> image or not?

Ah, now I see your point.

As I understand it, Alex's point is to introduce an additional, breve, binary
porcelain for the user. Adding or removing hairy IMAGE_INSTALL lines should be
hidden in the plumbing.

E.g., poky/meta/recipes-sato/images/core-image-sato.bb:

IMAGE_FEATURES += "splash package-management x11-base x11-sato \
ssh-server-dropbear hwcodecs"

When we have that many optional things, copy-pasting them in multiple recipes
would be an error-prone process and a maintenance problem due to duplication of
IMAGE_INSTALL += lines that have all to be updated if anything changes.

In our case, local.conf would have IMAGE_FEATURES = "kernel" by default. Of
course, we can only go in that direction if something like
IMAGE_FEATURES_isar-image-base = "" would work (maybe it does [1]). It isn't
very Isar-like, since Yocto is focused on one config at a time, and we
explicitly aim at multiple, diverse deliverables from the beginning. Ideally,
I'd like to see such binary porcelain in a recipe (not touching machine and
distro configs being the goal).


> >>   - Cleanup steps like localepurge, and apt-get clean
> >
> >Just do it always, as in your patches?
> 
> You meant the '*-configscripts.sh'?
> 
> AFAIK the purpose is to try to move most of this stuff out of there and into
> a 'system-setup-package' of sorts. At lease for all of those things, that
> don't require doing it at the end of everything.

Agree with the latter (e.g., root password, hostname).

Do I understand correctly that localepurge and apt-get clean have to be done at
the end? If yes, then they aren't suitable for system-setup-package. Then,
where should those go?

Regarding whether that should be done in IMAGE_FEATURES, my opinion is still
that specifically localepurge, apt-get clean should always be performed. IOW, I
don't think they belong to IMAGE_FEATURES.


> The rest of those steps (like those cleanup commands) should be split up
> into smaller scripts each with just one task and then trigger them via
> IMAGE_FEATURES after all packages were installed. Then the difference
> between raspbian and debian might just be some different entries in the
> IMAGE_FEATURES (or DISTRO_FEATURES) variable. At least thats my current idea
> here.

We can discuss those when we have a specific example.


> >>   - Removal of apt and other possible reduction tasks
> >
> >Additional recipe some images depend on?
> 
> Removal of apt and other reduction steps should be done after all packages
> are installed using apt. Doing that in some other recipe/debian package just
> doesn't work.

Ok, so it has to be a task. Yocto does it by default and provides
IMAGE_FEATURES += "package-management". I think mimicking that would be an
advantage here.


> >>The other question might be if having a kernel should be implemented as a
> >>IMAGE_FEATURE or rather a setting in the machine.conf?

I'd like to provide a binary way to customize it for every image without
touching machine or distro confs.


> >We already have it in IMAGE_INSTALL for isar-image-base. Doesn't it work if the
> >kernel is removed from there?
> 
> This is how I do it in the current "Optional kernel" patchset.

After I see the implementation, I think stacking complex lines in multiple
images has disadvantages compared to binary options.


References:

1. https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html#conditional-syntax-overrides


With kind regards,
Baurzhan.

Re: 'IMAGE_FEATURES' in Isar

[Claudius Heine]

Hi Baurzhan,

On 2018-04-13 13:27, Baurzhan Ismagulov wrote:
> On Thu, Apr 12, 2018 at 05:35:17PM +0200, Claudius Heine wrote:
>> Alex brought up the IMAGE_FEATURES idea and this thread just discuss it and
>> to answer the questions: What else could IMAGE_FEATURES be used for? And
>> does it make sense to use it to specify if the kernel should be part of the
>> image or not?
> 
> Ah, now I see your point.
> 
> As I understand it, Alex's point is to introduce an additional, breve, binary
> porcelain for the user. Adding or removing hairy IMAGE_INSTALL lines should be
> hidden in the plumbing.
> 
> E.g., poky/meta/recipes-sato/images/core-image-sato.bb:
> 
> IMAGE_FEATURES += "splash package-management x11-base x11-sato \
> ssh-server-dropbear hwcodecs"
> 
> When we have that many optional things, copy-pasting them in multiple recipes
> would be an error-prone process and a maintenance problem due to duplication of
> IMAGE_INSTALL += lines that have all to be updated if anything changes.
> 
> In our case, local.conf would have IMAGE_FEATURES = "kernel" by default. Of
> course, we can only go in that direction if something like
> IMAGE_FEATURES_isar-image-base = "" would work (maybe it does [1]). It isn't
> very Isar-like, since Yocto is focused on one config at a time, and we
> explicitly aim at multiple, diverse deliverables from the beginning. Ideally,
> I'd like to see such binary porcelain in a recipe (not touching machine and
> distro configs being the goal).

The problem with porting 'IMAGE_FEATURES' directly to Isar can be read 
in the OE documentation:

     IMAGE_FEATURES: Specifies features to include in the image. Most of
                     these features map to additional packages for
                     installation. [1]

Of course IMAGE_FEATURES in OE do a bit more that just 'IMAGE_INSTALL += 
"packagegroup-feature"' or just 'IMAGE_INSTALL += apt'. But that would 
be the case if we start mapping those features directly to Isar, since 
using a binary distribution offers less customizability than using a 
source distribution.

The basic idea of IMAGE_FEATURES is to be orthogonal to IMAGE_INSTALL, 
otherwise having it doesn't make sense. You have to use it for stuff you 
cannot express via just some entries in IMAGE_INSTALL. The only thing 
that comes to my mind and is not covered in some other aspect of Isar 
already, are those currently monolithic configuration scripts.

Breaking those scripts up into smaller parts like 
'no-package-management', 'no-headers', 'no-manpages', 
'no-other-locales', 'no-package-cache' and maybe some other things I 
currently cannot think of, and executing them after all packages are 
install, controlled by IMAGE_FEATURES, would be orthogonal to 
IMAGE_INSTALL entries and would provide value.

Yes, those 'no-*' features aren't OE like, but this is a concession to 
the binary distribution Isar is based on.

Also to the idea of OVERRRIDES, we currently don't use them at all, and 
we should add at least the MACHINE, DISTRO, DISTRO_ARCH and MULTICONFIG 
to it, just to be able to better select different configurations in 
bbappend files. About your idea with adding the image recipe name to it, 
I don't know if that is even possible.

>>>>    - Cleanup steps like localepurge, and apt-get clean
>>>
>>> Just do it always, as in your patches?
>>
>> You meant the '*-configscripts.sh'?
>>
>> AFAIK the purpose is to try to move most of this stuff out of there and into
>> a 'system-setup-package' of sorts. At lease for all of those things, that
>> don't require doing it at the end of everything.
> 
> Agree with the latter (e.g., root password, hostname).
> 
> Do I understand correctly that localepurge and apt-get clean have to be done at
> the end? If yes, then they aren't suitable for system-setup-package. Then,
> where should those go?

As I described above: Into IMAGE_FEATURES, since those steps are 
orthogonal to IMAGE_INSTALL or packages in general.

> Regarding whether that should be done in IMAGE_FEATURES, my opinion is still
> that specifically localepurge, apt-get clean should always be performed. IOW, I
> don't think they belong to IMAGE_FEATURES.

I am not so sure about that. For instance the 'buildchroot' is also 
basically just an image, where those steps aren't necessary. But maybe 
deploying the 'build.sh' script is a IMAGE_FEATURE for buildchroot. We 
cannot install it as a package, since the required buildchroot for 
packaging it is not done yet.

>> The rest of those steps (like those cleanup commands) should be split up
>> into smaller scripts each with just one task and then trigger them via
>> IMAGE_FEATURES after all packages were installed. Then the difference
>> between raspbian and debian might just be some different entries in the
>> IMAGE_FEATURES (or DISTRO_FEATURES) variable. At least thats my current idea
>> here.
> 
> We can discuss those when we have a specific example.

I just took a look at what the current configscripts are doing. Here are 
my basic thoughts:

   - install /etc/default/locales  > package
   - configuration of localepurge  > package
   - debconf settings              > Don't know yet, maybe package?
   - install /etc/fstab            > package or wic
   - create devices                > package
   - /etc/inittab                  > package
   - execute localepurge           > feature that also depends on package
   - apt-get clean                 > feature
   - setting boot cfg              > package or wic
   - remove qemu-*-static          > feature (since buildchroot doesn't 
need it)

> 
> 
>>>>    - Removal of apt and other possible reduction tasks
>>>
>>> Additional recipe some images depend on?
>>
>> Removal of apt and other reduction steps should be done after all packages
>> are installed using apt. Doing that in some other recipe/debian package just
>> doesn't work.
> 
> Ok, so it has to be a task. Yocto does it by default and provides
> IMAGE_FEATURES += "package-management". I think mimicking that would be an
> advantage here.

As I said further up: OE can do stuff like 'package-management' feature 
because they start with a completly clean state and specifying only 
stuff to add is possible there. With Isar we start with a minimal base 
system, that already includes a package-managment per default. A feature 
to have it doesn't make sense, a feature to remove it does IMO.

>>>> The other question might be if having a kernel should be implemented as a
>>>> IMAGE_FEATURE or rather a setting in the machine.conf?
> 
> I'd like to provide a binary way to customize it for every image without
> touching machine or distro confs.

Ok. I get it. From were I came from, I used 'having a kernel' and 'not 
having a kernel' as a function of the machine configuration or 
multiconfig configuration. But you see it as a function of the image.

Since adding the kernel to IMAGE_INSTALL or evaluating IMAGE_FEATURES is 
mostly done in the specific image, it can also be changed there.

>>> We already have it in IMAGE_INSTALL for isar-image-base. Doesn't it work if the
>>> kernel is removed from there?
>>
>> This is how I do it in the current "Optional kernel" patchset.
> 
> After I see the implementation, I think stacking complex lines in multiple
> images has disadvantages compared to binary options.

I don't think I understand you here.

Do you mean that you prefer having:

     IMAGE_INSTALL += "${@ ("linux-image-" + d.getVar("KERNEL_NAME", 
True)) if 'kernel' in d.getVar("IMAGE_FEATURES", True) else ""}"

instead of:

     IMAGE_INSTALL += "${@ ("linux-image-" + d.getVar("KERNEL_NAME", 
True)) if d.getVar("KERNEL_NAME", True) else ""}"

If that is your issue, than ok, we can change that. But as I said, I 
don't think that IMAGE_FEATURES should work that way, since that way 
they are just a simple mapping of string -> string. Meaning a string in 
IMAGE_FEATURES results in a string in IMAGE_INSTALL. And we miss out of 
using IMAGE_FEATURES in a way orthogonal to IMAGE_INSTALL.

Cheers,
Claudius

[1] 
https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#image-generation-dev-environment

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: 'IMAGE_FEATURES' in Isar

[Claudius Heine]

Hi again,

On 2018-04-16 10:17, [ext] Claudius Heine wrote:
> 
> Also to the idea of OVERRRIDES, we currently don't use them at all, and 
> we should add at least the MACHINE, DISTRO, DISTRO_ARCH and MULTICONFIG 
> to it, just to be able to better select different configurations in 
> bbappend files. About your idea with adding the image recipe name to it, 
> I don't know if that is even possible.

I think I now know what you meat by 'IMAGE_FEATURES_isar-image-base = 
""'. You meant adding that to a configuration file to overwrite certain 
variables from outside the recipe. Yes that could be possible. I don't 
do that very much, I rather write bbappends for those cases.

Cheers,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de