[PATCH v2] meta/classes/base: Fix showdata and liststasks after bitbake update

[PATCH v2] meta/classes/base: Fix showdata and liststasks after bitbake update

[Alexander Smirnov]

From: Claudius Heine <ch@denx.de>

8<--

Just added references to origin code in comment. Didn't touch the code.

8<--

Those tasks were broken, this patchs fixes by implementing them similar
to how openembedded-core does it.

Signed-off-by: Claudius Heine <ch@denx.de>
---
 meta/classes/base.bbclass | 30 +++++++++++++++++++-----------
 1 file changed, 19 insertions(+), 11 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 7d44f72..ae84677 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -37,25 +37,33 @@ bbfatal() {
 	exit 1
 }
 
+# Derived from bitbake: bitbake/classes/base.bbclass
 addtask showdata
 do_showdata[nostamp] = "1"
 python do_showdata() {
-	import sys
-	# emit variables and shell functions
-	bb.data.emit_env(sys.__stdout__, d, True)
-	# emit the metadata which isnt valid shell
-	for e in bb.data.keys(d):
-		if bb.data.getVarFlag(e, 'python', d):
-			sys.__stdout__.write("\npython %s () {\n%s}\n" % (e, bb.data.getVar(e, d, 1)))
+    for e in d.keys():
+        if d.getVarFlag(e, 'python'):
+            bb.plain("\npython %s () {\n%s}\n" % (e, d.getVar(e, True)))
 }
 
+# Derived from Open Embedded: openembedded-core/meta/classes/utility-tasks.bbclass
 addtask listtasks
 do_listtasks[nostamp] = "1"
 python do_listtasks() {
-	import sys
-	for e in bb.data.keys(d):
-		if bb.data.getVarFlag(e, 'task', d):
-			sys.__stdout__.write("%s\n" % e)
+    taskdescs = {}
+    maxlen = 0
+    for e in d.keys():
+        if d.getVarFlag(e, 'task'):
+            maxlen = max(maxlen, len(e))
+            if e.endswith('_setscene'):
+                desc = "%s (setscene version)" % (d.getVarFlag(e[:-9], 'doc') or '')
+            else:
+                desc = d.getVarFlag(e, 'doc') or ''
+            taskdescs[e] = desc
+
+    tasks = sorted(taskdescs.keys())
+    for taskname in tasks:
+        bb.plain("%s  %s" % (taskname.ljust(maxlen), taskdescs[taskname]))
 }
 
 do_fetch[dirs] = "${DL_DIR}"
-- 
2.1.4

Re: [PATCH v2] meta/classes/base: Fix showdata and liststasks after bitbake update

[Henning Schild]

Mhh this one rings a bell ;)

https://groups.google.com/d/msg/isar-users/Raq3ayrqtnc/oqTt6_sSBQAJ

Someone has to reread that and check whether my symlink approach was
rejected rightfully, or whether it was just one of the many
misunderstandings we had in the past.

Henning

Am Fri, 9 Feb 2018 20:28:50 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> From: Claudius Heine <ch@denx.de>
> 
> 8<--
> 
> Just added references to origin code in comment. Didn't touch the
> code.
> 
> 8<--
> 
> Those tasks were broken, this patchs fixes by implementing them
> similar to how openembedded-core does it.
> 
> Signed-off-by: Claudius Heine <ch@denx.de>
> ---
>  meta/classes/base.bbclass | 30 +++++++++++++++++++-----------
>  1 file changed, 19 insertions(+), 11 deletions(-)
> 
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index 7d44f72..ae84677 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -37,25 +37,33 @@ bbfatal() {
>  	exit 1
>  }
>  
> +# Derived from bitbake: bitbake/classes/base.bbclass
>  addtask showdata
>  do_showdata[nostamp] = "1"
>  python do_showdata() {
> -	import sys
> -	# emit variables and shell functions
> -	bb.data.emit_env(sys.__stdout__, d, True)
> -	# emit the metadata which isnt valid shell
> -	for e in bb.data.keys(d):
> -		if bb.data.getVarFlag(e, 'python', d):
> -			sys.__stdout__.write("\npython %s ()
> {\n%s}\n" % (e, bb.data.getVar(e, d, 1)))
> +    for e in d.keys():
> +        if d.getVarFlag(e, 'python'):
> +            bb.plain("\npython %s () {\n%s}\n" % (e, d.getVar(e,
> True))) }
>  
> +# Derived from Open Embedded:
> openembedded-core/meta/classes/utility-tasks.bbclass addtask listtasks
>  do_listtasks[nostamp] = "1"
>  python do_listtasks() {
> -	import sys
> -	for e in bb.data.keys(d):
> -		if bb.data.getVarFlag(e, 'task', d):
> -			sys.__stdout__.write("%s\n" % e)
> +    taskdescs = {}
> +    maxlen = 0
> +    for e in d.keys():
> +        if d.getVarFlag(e, 'task'):
> +            maxlen = max(maxlen, len(e))
> +            if e.endswith('_setscene'):
> +                desc = "%s (setscene version)" %
> (d.getVarFlag(e[:-9], 'doc') or '')
> +            else:
> +                desc = d.getVarFlag(e, 'doc') or ''
> +            taskdescs[e] = desc
> +
> +    tasks = sorted(taskdescs.keys())
> +    for taskname in tasks:
> +        bb.plain("%s  %s" % (taskname.ljust(maxlen),
> taskdescs[taskname])) }
>  
>  do_fetch[dirs] = "${DL_DIR}"

Re: [PATCH v2] meta/classes/base: Fix showdata and liststasks after bitbake update

[Alexander Smirnov]

Henning Schild <henning.schild@siemens.com> 9 февраля 2018 г. 21:16:17 написал:

> Mhh this one rings a bell ;)
>
> https://groups.google.com/d/msg/isar-users/Raq3ayrqtnc/oqTt6_sSBQAJ
>
> Someone has to reread that and check whether my symlink approach was
> rejected rightfully, or whether it was just one of the many
> misunderstandings we had in the past.

This approach was rejected due to several reasons:

 - According to the bitbake documentation, you need to create your own 
base.bbclass. I consider this as a signal - do not reply on default one.

https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html#hello-world-example

 - Default base.bbclass provides inevitable default settings that should be 
overwritten in custom class. Moreover this class should be included to 
*all* the Isar recipes manually. Inherit something to overwrite it 
immediately still doesn't look reasonable for me.

In addition:

 - do_listtasks in this patch seems to be taken from oe-core, not from 
bitbake, so inheritance doesn't help here.

Alex

>
> Henning
>
> Am Fri, 9 Feb 2018 20:28:50 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>
>> From: Claudius Heine <ch@denx.de>
>>
>> 8<--
>>
>> Just added references to origin code in comment. Didn't touch the
>> code.
>>
>> 8<--
>>
>> Those tasks were broken, this patchs fixes by implementing them
>> similar to how openembedded-core does it.
>>
>> Signed-off-by: Claudius Heine <ch@denx.de>
>> ---
>>  meta/classes/base.bbclass | 30 +++++++++++++++++++-----------
>>  1 file changed, 19 insertions(+), 11 deletions(-)
>>
>> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
>> index 7d44f72..ae84677 100644
>> --- a/meta/classes/base.bbclass
>> +++ b/meta/classes/base.bbclass
>> @@ -37,25 +37,33 @@ bbfatal() {
>>  	exit 1
>>  }
>>
>> +# Derived from bitbake: bitbake/classes/base.bbclass
>>  addtask showdata
>>  do_showdata[nostamp] = "1"
>>  python do_showdata() {
>> -	import sys
>> -	# emit variables and shell functions
>> -	bb.data.emit_env(sys.__stdout__, d, True)
>> -	# emit the metadata which isnt valid shell
>> -	for e in bb.data.keys(d):
>> -		if bb.data.getVarFlag(e, 'python', d):
>> -			sys.__stdout__.write("\npython %s ()
>> {\n%s}\n" % (e, bb.data.getVar(e, d, 1)))
>> +    for e in d.keys():
>> +        if d.getVarFlag(e, 'python'):
>> +            bb.plain("\npython %s () {\n%s}\n" % (e, d.getVar(e,
>> True))) }
>>
>> +# Derived from Open Embedded:
>> openembedded-core/meta/classes/utility-tasks.bbclass addtask listtasks
>>  do_listtasks[nostamp] = "1"
>>  python do_listtasks() {
>> -	import sys
>> -	for e in bb.data.keys(d):
>> -		if bb.data.getVarFlag(e, 'task', d):
>> -			sys.__stdout__.write("%s\n" % e)
>> +    taskdescs = {}
>> +    maxlen = 0
>> +    for e in d.keys():
>> +        if d.getVarFlag(e, 'task'):
>> +            maxlen = max(maxlen, len(e))
>> +            if e.endswith('_setscene'):
>> +                desc = "%s (setscene version)" %
>> (d.getVarFlag(e[:-9], 'doc') or '')
>> +            else:
>> +                desc = d.getVarFlag(e, 'doc') or ''
>> +            taskdescs[e] = desc
>> +
>> +    tasks = sorted(taskdescs.keys())
>> +    for taskname in tasks:
>> +        bb.plain("%s  %s" % (taskname.ljust(maxlen),
>> taskdescs[taskname])) }
>>
>>  do_fetch[dirs] = "${DL_DIR}"
>
--
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

Re: [PATCH v2] meta/classes/base: Fix showdata and liststasks after bitbake update

[Henning Schild]

Am Fri, 9 Feb 2018 21:42:12 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Henning Schild <henning.schild@siemens.com> 9 февраля 2018 г.
> 21:16:17 написал:
> 
> > Mhh this one rings a bell ;)
> >
> > https://groups.google.com/d/msg/isar-users/Raq3ayrqtnc/oqTt6_sSBQAJ
> >
> > Someone has to reread that and check whether my symlink approach was
> > rejected rightfully, or whether it was just one of the many
> > misunderstandings we had in the past.  
> 
> This approach was rejected due to several reasons:
> 
>  - According to the bitbake documentation, you need to create your
> own base.bbclass. I consider this as a signal - do not reply on
> default one.
> 
> https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html#hello-world-example
> 
>  - Default base.bbclass provides inevitable default settings that
> should be overwritten in custom class. Moreover this class should be
> included to *all* the Isar recipes manually. Inherit something to
> overwrite it immediately still doesn't look reasonable for me.
> 
> In addition:
> 
>  - do_listtasks in this patch seems to be taken from oe-core, not
> from bitbake, so inheritance doesn't help here.

Ok, thanks for summing it up again. It might be a good idea to write a
developer doc on how to upgrade bitbake (and wic) and what to test when
doing so. Otherwise we risk introducing the same problems over and over
again.
But that could also be a testcase.

Henning

> Alex
> 
> >
> > Henning
> >
> > Am Fri, 9 Feb 2018 20:28:50 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >  
> >> From: Claudius Heine <ch@denx.de>
> >>
> >> 8<--
> >>
> >> Just added references to origin code in comment. Didn't touch the
> >> code.
> >>
> >> 8<--
> >>
> >> Those tasks were broken, this patchs fixes by implementing them
> >> similar to how openembedded-core does it.
> >>
> >> Signed-off-by: Claudius Heine <ch@denx.de>
> >> ---
> >>  meta/classes/base.bbclass | 30 +++++++++++++++++++-----------
> >>  1 file changed, 19 insertions(+), 11 deletions(-)
> >>
> >> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> >> index 7d44f72..ae84677 100644
> >> --- a/meta/classes/base.bbclass
> >> +++ b/meta/classes/base.bbclass
> >> @@ -37,25 +37,33 @@ bbfatal() {
> >>  	exit 1
> >>  }
> >>
> >> +# Derived from bitbake: bitbake/classes/base.bbclass
> >>  addtask showdata
> >>  do_showdata[nostamp] = "1"
> >>  python do_showdata() {
> >> -	import sys
> >> -	# emit variables and shell functions
> >> -	bb.data.emit_env(sys.__stdout__, d, True)
> >> -	# emit the metadata which isnt valid shell
> >> -	for e in bb.data.keys(d):
> >> -		if bb.data.getVarFlag(e, 'python', d):
> >> -			sys.__stdout__.write("\npython %s ()
> >> {\n%s}\n" % (e, bb.data.getVar(e, d, 1)))
> >> +    for e in d.keys():
> >> +        if d.getVarFlag(e, 'python'):
> >> +            bb.plain("\npython %s () {\n%s}\n" % (e, d.getVar(e,
> >> True))) }
> >>
> >> +# Derived from Open Embedded:
> >> openembedded-core/meta/classes/utility-tasks.bbclass addtask
> >> listtasks do_listtasks[nostamp] = "1"
> >>  python do_listtasks() {
> >> -	import sys
> >> -	for e in bb.data.keys(d):
> >> -		if bb.data.getVarFlag(e, 'task', d):
> >> -			sys.__stdout__.write("%s\n" % e)
> >> +    taskdescs = {}
> >> +    maxlen = 0
> >> +    for e in d.keys():
> >> +        if d.getVarFlag(e, 'task'):
> >> +            maxlen = max(maxlen, len(e))
> >> +            if e.endswith('_setscene'):
> >> +                desc = "%s (setscene version)" %
> >> (d.getVarFlag(e[:-9], 'doc') or '')
> >> +            else:
> >> +                desc = d.getVarFlag(e, 'doc') or ''
> >> +            taskdescs[e] = desc
> >> +
> >> +    tasks = sorted(taskdescs.keys())
> >> +    for taskname in tasks:
> >> +        bb.plain("%s  %s" % (taskname.ljust(maxlen),
> >> taskdescs[taskname])) }
> >>
> >>  do_fetch[dirs] = "${DL_DIR}"  
> >  
> --
> With best regards,
> Alexander Smirnov
> 
> ilbers GmbH
> Baierbrunner Str. 28c
> D-81379 Munich
> +49 (89) 122 67 24-0
> http://ilbers.de/
> Commercial register Munich, HRB 214197
> General manager: Baurzhan Ismagulov
> 
> 

Re: [PATCH v2] meta/classes/base: Fix showdata and liststasks after bitbake update

[Alexander Smirnov]

Henning Schild <henning.schild@siemens.com> 9 февраля 2018 г. 22:33:48 написал:

> Am Fri, 9 Feb 2018 21:42:12 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>
>> Henning Schild <henning.schild@siemens.com> 9 февраля 2018 г.
>> 21:16:17 написал:
>>
>> > Mhh this one rings a bell ;)
>> >
>> > https://groups.google.com/d/msg/isar-users/Raq3ayrqtnc/oqTt6_sSBQAJ
>> >
>> > Someone has to reread that and check whether my symlink approach was
>> > rejected rightfully, or whether it was just one of the many
>> > misunderstandings we had in the past.
>>
>> This approach was rejected due to several reasons:
>>
>>  - According to the bitbake documentation, you need to create your
>> own base.bbclass. I consider this as a signal - do not reply on
>> default one.
>>
>> https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html#hello-world-example
>>
>>  - Default base.bbclass provides inevitable default settings that
>> should be overwritten in custom class. Moreover this class should be
>> included to *all* the Isar recipes manually. Inherit something to
>> overwrite it immediately still doesn't look reasonable for me.
>>
>> In addition:
>>
>>  - do_listtasks in this patch seems to be taken from oe-core, not
>> from bitbake, so inheritance doesn't help here.
>
> Ok, thanks for summing it up again. It might be a good idea to write a
> developer doc on how to upgrade bitbake (and wic) and what to test when
> doing so. Otherwise we risk introducing the same problems over and over
> again.
> But that could also be a testcase.
>

Definitely agree. This was the first bitbake update and it brought several 
unexpected issues, that still appears.

I think it's a good time to summarize this experience and make the 
document/test. Also I'd like to consider git modules for bitbake, like Jan 
proposed. So I could do this.

Apart from above I see here several other topics:
 - How introduce 3rd party software. At the moment we have bitbake and wic. 
The last one is derived from OE, so do we need to mention this explicitly?
 - I assume that this patch is not the last one, which brings pieces of 
code from other projects. How this code should be marked? In the top in the 
file's header? Inline in the file?
 - Jan already proposed to define unified Isar header with copyrights. For 
example our base.class has no Isar copyright at all.

Regarding bitbake updates, seems Yocto and OE handle this manually, the 
have their own set of classes which updated according to the current 
bitbake version.

Alex

> Henning
>
>> Alex
>>
>> >
>> > Henning
>> >
>> > Am Fri, 9 Feb 2018 20:28:50 +0300
>> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>> >
>> >> From: Claudius Heine <ch@denx.de>
>> >>
>> >> 8<--
>> >>
>> >> Just added references to origin code in comment. Didn't touch the
>> >> code.
>> >>
>> >> 8<--
>> >>
>> >> Those tasks were broken, this patchs fixes by implementing them
>> >> similar to how openembedded-core does it.
>> >>
>> >> Signed-off-by: Claudius Heine <ch@denx.de>
>> >> ---
>> >>  meta/classes/base.bbclass | 30 +++++++++++++++++++-----------
>> >>  1 file changed, 19 insertions(+), 11 deletions(-)
>> >>
>> >> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
>> >> index 7d44f72..ae84677 100644
>> >> --- a/meta/classes/base.bbclass
>> >> +++ b/meta/classes/base.bbclass
>> >> @@ -37,25 +37,33 @@ bbfatal() {
>> >>  	exit 1
>> >>  }
>> >>
>> >> +# Derived from bitbake: bitbake/classes/base.bbclass
>> >>  addtask showdata
>> >>  do_showdata[nostamp] = "1"
>> >>  python do_showdata() {
>> >> -	import sys
>> >> -	# emit variables and shell functions
>> >> -	bb.data.emit_env(sys.__stdout__, d, True)
>> >> -	# emit the metadata which isnt valid shell
>> >> -	for e in bb.data.keys(d):
>> >> -		if bb.data.getVarFlag(e, 'python', d):
>> >> -			sys.__stdout__.write("\npython %s ()
>> >> {\n%s}\n" % (e, bb.data.getVar(e, d, 1)))
>> >> +    for e in d.keys():
>> >> +        if d.getVarFlag(e, 'python'):
>> >> +            bb.plain("\npython %s () {\n%s}\n" % (e, d.getVar(e,
>> >> True))) }
>> >>
>> >> +# Derived from Open Embedded:
>> >> openembedded-core/meta/classes/utility-tasks.bbclass addtask
>> >> listtasks do_listtasks[nostamp] = "1"
>> >>  python do_listtasks() {
>> >> -	import sys
>> >> -	for e in bb.data.keys(d):
>> >> -		if bb.data.getVarFlag(e, 'task', d):
>> >> -			sys.__stdout__.write("%s\n" % e)
>> >> +    taskdescs = {}
>> >> +    maxlen = 0
>> >> +    for e in d.keys():
>> >> +        if d.getVarFlag(e, 'task'):
>> >> +            maxlen = max(maxlen, len(e))
>> >> +            if e.endswith('_setscene'):
>> >> +                desc = "%s (setscene version)" %
>> >> (d.getVarFlag(e[:-9], 'doc') or '')
>> >> +            else:
>> >> +                desc = d.getVarFlag(e, 'doc') or ''
>> >> +            taskdescs[e] = desc
>> >> +
>> >> +    tasks = sorted(taskdescs.keys())
>> >> +    for taskname in tasks:
>> >> +        bb.plain("%s  %s" % (taskname.ljust(maxlen),
>> >> taskdescs[taskname])) }
>> >>
>> >>  do_fetch[dirs] = "${DL_DIR}"
>> >
>> --
>> With best regards,
>> Alexander Smirnov
>>
>> ilbers GmbH
>> Baierbrunner Str. 28c
>> D-81379 Munich
>> +49 (89) 122 67 24-0
>> http://ilbers.de/
>> Commercial register Munich, HRB 214197
>> General manager: Baurzhan Ismagulov
>>
>>
>
--
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

Re: [PATCH v2] meta/classes/base: Fix showdata and liststasks after bitbake update

[Jan Kiszka]

On 2018-02-09 18:28, Alexander Smirnov wrote:
> From: Claudius Heine <ch@denx.de>
> 
> 8<--
> 
> Just added references to origin code in comment. Didn't touch the code.
> 
> 8<--
> 
> Those tasks were broken, this patchs fixes by implementing them similar
> to how openembedded-core does it.
> 
> Signed-off-by: Claudius Heine <ch@denx.de>
> ---
>  meta/classes/base.bbclass | 30 +++++++++++++++++++-----------
>  1 file changed, 19 insertions(+), 11 deletions(-)
> 
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index 7d44f72..ae84677 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -37,25 +37,33 @@ bbfatal() {
>  	exit 1
>  }
>  
> +# Derived from bitbake: bitbake/classes/base.bbclass
>  addtask showdata
>  do_showdata[nostamp] = "1"
>  python do_showdata() {
> -	import sys
> -	# emit variables and shell functions
> -	bb.data.emit_env(sys.__stdout__, d, True)
> -	# emit the metadata which isnt valid shell
> -	for e in bb.data.keys(d):
> -		if bb.data.getVarFlag(e, 'python', d):
> -			sys.__stdout__.write("\npython %s () {\n%s}\n" % (e, bb.data.getVar(e, d, 1)))
> +    for e in d.keys():
> +        if d.getVarFlag(e, 'python'):
> +            bb.plain("\npython %s () {\n%s}\n" % (e, d.getVar(e, True)))
>  }
>  
> +# Derived from Open Embedded: openembedded-core/meta/classes/utility-tasks.bbclass
>  addtask listtasks
>  do_listtasks[nostamp] = "1"
>  python do_listtasks() {
> -	import sys
> -	for e in bb.data.keys(d):
> -		if bb.data.getVarFlag(e, 'task', d):
> -			sys.__stdout__.write("%s\n" % e)
> +    taskdescs = {}
> +    maxlen = 0
> +    for e in d.keys():
> +        if d.getVarFlag(e, 'task'):
> +            maxlen = max(maxlen, len(e))
> +            if e.endswith('_setscene'):
> +                desc = "%s (setscene version)" % (d.getVarFlag(e[:-9], 'doc') or '')
> +            else:
> +                desc = d.getVarFlag(e, 'doc') or ''
> +            taskdescs[e] = desc
> +
> +    tasks = sorted(taskdescs.keys())
> +    for taskname in tasks:
> +        bb.plain("%s  %s" % (taskname.ljust(maxlen), taskdescs[taskname]))
>  }
>  
>  do_fetch[dirs] = "${DL_DIR}"
> 

Looks good to me, and it also fixes the broken indention of those tasks,
something bitbake started to complain here after enabling signatures.
Please merge to next.

Jan

Re: [PATCH v2] meta/classes/base: Fix showdata and liststasks after bitbake update

[Alexander Smirnov]

> Looks good to me, and it also fixes the broken indention of those tasks,
> something bitbake started to complain here after enabling signatures.
> Please merge to next.

Applied.

Alex