Re: 'IMAGE_FEATURES' in Isar

[Claudius Heine <claudius.heine.ext@siemens.com>]

Hi Baurzhan,

On 2018-04-13 13:27, Baurzhan Ismagulov wrote:
> On Thu, Apr 12, 2018 at 05:35:17PM +0200, Claudius Heine wrote:
>> Alex brought up the IMAGE_FEATURES idea and this thread just discuss it and
>> to answer the questions: What else could IMAGE_FEATURES be used for? And
>> does it make sense to use it to specify if the kernel should be part of the
>> image or not?
> 
> Ah, now I see your point.
> 
> As I understand it, Alex's point is to introduce an additional, breve, binary
> porcelain for the user. Adding or removing hairy IMAGE_INSTALL lines should be
> hidden in the plumbing.
> 
> E.g., poky/meta/recipes-sato/images/core-image-sato.bb:
> 
> IMAGE_FEATURES += "splash package-management x11-base x11-sato \
> ssh-server-dropbear hwcodecs"
> 
> When we have that many optional things, copy-pasting them in multiple recipes
> would be an error-prone process and a maintenance problem due to duplication of
> IMAGE_INSTALL += lines that have all to be updated if anything changes.
> 
> In our case, local.conf would have IMAGE_FEATURES = "kernel" by default. Of
> course, we can only go in that direction if something like
> IMAGE_FEATURES_isar-image-base = "" would work (maybe it does [1]). It isn't
> very Isar-like, since Yocto is focused on one config at a time, and we
> explicitly aim at multiple, diverse deliverables from the beginning. Ideally,
> I'd like to see such binary porcelain in a recipe (not touching machine and
> distro configs being the goal).

The problem with porting 'IMAGE_FEATURES' directly to Isar can be read 
in the OE documentation:

     IMAGE_FEATURES: Specifies features to include in the image. Most of
                     these features map to additional packages for
                     installation. [1]

Of course IMAGE_FEATURES in OE do a bit more that just 'IMAGE_INSTALL += 
"packagegroup-feature"' or just 'IMAGE_INSTALL += apt'. But that would 
be the case if we start mapping those features directly to Isar, since 
using a binary distribution offers less customizability than using a 
source distribution.

The basic idea of IMAGE_FEATURES is to be orthogonal to IMAGE_INSTALL, 
otherwise having it doesn't make sense. You have to use it for stuff you 
cannot express via just some entries in IMAGE_INSTALL. The only thing 
that comes to my mind and is not covered in some other aspect of Isar 
already, are those currently monolithic configuration scripts.

Breaking those scripts up into smaller parts like 
'no-package-management', 'no-headers', 'no-manpages', 
'no-other-locales', 'no-package-cache' and maybe some other things I 
currently cannot think of, and executing them after all packages are 
install, controlled by IMAGE_FEATURES, would be orthogonal to 
IMAGE_INSTALL entries and would provide value.

Yes, those 'no-*' features aren't OE like, but this is a concession to 
the binary distribution Isar is based on.

Also to the idea of OVERRRIDES, we currently don't use them at all, and 
we should add at least the MACHINE, DISTRO, DISTRO_ARCH and MULTICONFIG 
to it, just to be able to better select different configurations in 
bbappend files. About your idea with adding the image recipe name to it, 
I don't know if that is even possible.

>>>>    - Cleanup steps like localepurge, and apt-get clean
>>>
>>> Just do it always, as in your patches?
>>
>> You meant the '*-configscripts.sh'?
>>
>> AFAIK the purpose is to try to move most of this stuff out of there and into
>> a 'system-setup-package' of sorts. At lease for all of those things, that
>> don't require doing it at the end of everything.
> 
> Agree with the latter (e.g., root password, hostname).
> 
> Do I understand correctly that localepurge and apt-get clean have to be done at
> the end? If yes, then they aren't suitable for system-setup-package. Then,
> where should those go?

As I described above: Into IMAGE_FEATURES, since those steps are 
orthogonal to IMAGE_INSTALL or packages in general.

> Regarding whether that should be done in IMAGE_FEATURES, my opinion is still
> that specifically localepurge, apt-get clean should always be performed. IOW, I
> don't think they belong to IMAGE_FEATURES.

I am not so sure about that. For instance the 'buildchroot' is also 
basically just an image, where those steps aren't necessary. But maybe 
deploying the 'build.sh' script is a IMAGE_FEATURE for buildchroot. We 
cannot install it as a package, since the required buildchroot for 
packaging it is not done yet.

>> The rest of those steps (like those cleanup commands) should be split up
>> into smaller scripts each with just one task and then trigger them via
>> IMAGE_FEATURES after all packages were installed. Then the difference
>> between raspbian and debian might just be some different entries in the
>> IMAGE_FEATURES (or DISTRO_FEATURES) variable. At least thats my current idea
>> here.
> 
> We can discuss those when we have a specific example.

I just took a look at what the current configscripts are doing. Here are 
my basic thoughts:

   - install /etc/default/locales  > package
   - configuration of localepurge  > package
   - debconf settings              > Don't know yet, maybe package?
   - install /etc/fstab            > package or wic
   - create devices                > package
   - /etc/inittab                  > package
   - execute localepurge           > feature that also depends on package
   - apt-get clean                 > feature
   - setting boot cfg              > package or wic
   - remove qemu-*-static          > feature (since buildchroot doesn't 
need it)

> 
> 
>>>>    - Removal of apt and other possible reduction tasks
>>>
>>> Additional recipe some images depend on?
>>
>> Removal of apt and other reduction steps should be done after all packages
>> are installed using apt. Doing that in some other recipe/debian package just
>> doesn't work.
> 
> Ok, so it has to be a task. Yocto does it by default and provides
> IMAGE_FEATURES += "package-management". I think mimicking that would be an
> advantage here.

As I said further up: OE can do stuff like 'package-management' feature 
because they start with a completly clean state and specifying only 
stuff to add is possible there. With Isar we start with a minimal base 
system, that already includes a package-managment per default. A feature 
to have it doesn't make sense, a feature to remove it does IMO.

>>>> The other question might be if having a kernel should be implemented as a
>>>> IMAGE_FEATURE or rather a setting in the machine.conf?
> 
> I'd like to provide a binary way to customize it for every image without
> touching machine or distro confs.

Ok. I get it. From were I came from, I used 'having a kernel' and 'not 
having a kernel' as a function of the machine configuration or 
multiconfig configuration. But you see it as a function of the image.

Since adding the kernel to IMAGE_INSTALL or evaluating IMAGE_FEATURES is 
mostly done in the specific image, it can also be changed there.

>>> We already have it in IMAGE_INSTALL for isar-image-base. Doesn't it work if the
>>> kernel is removed from there?
>>
>> This is how I do it in the current "Optional kernel" patchset.
> 
> After I see the implementation, I think stacking complex lines in multiple
> images has disadvantages compared to binary options.

I don't think I understand you here.

Do you mean that you prefer having:

     IMAGE_INSTALL += "${@ ("linux-image-" + d.getVar("KERNEL_NAME", 
True)) if 'kernel' in d.getVar("IMAGE_FEATURES", True) else ""}"

instead of:

     IMAGE_INSTALL += "${@ ("linux-image-" + d.getVar("KERNEL_NAME", 
True)) if d.getVar("KERNEL_NAME", True) else ""}"

If that is your issue, than ok, we can change that. But as I said, I 
don't think that IMAGE_FEATURES should work that way, since that way 
they are just a simple mapping of string -> string. Meaning a string in 
IMAGE_FEATURES results in a string in IMAGE_INSTALL. And we miss out of 
using IMAGE_FEATURES in a way orthogonal to IMAGE_INSTALL.

Cheers,
Claudius

[1] 
https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#image-generation-dev-environment

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de