[PATCH] utils: Ensure shell function failure in python logging is correct

[PATCH] utils: Ensure shell function failure in python logging is correct

[Anton Mikanovich]

If a python function exec_func() calls a shell task, the logging wasn't working
correctly in all cases since the exception was turned into a BBHandledException()
and the logfile piece was lost which is handled at the top task level.

The easiest way to avoid this is to allow the ExecutionError exceptions to
be raised to a higher level, we don't need the traceback for them.

Backported from Bitbake 2.0 upstream branch:
https://git.openembedded.org/bitbake/commit/?id=7cae11f558f9ff5fd05ef23b789aaef92fb5a327

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 bitbake/lib/bb/utils.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
index 2a150fe..74cbc69 100644
--- a/bitbake/lib/bb/utils.py
+++ b/bitbake/lib/bb/utils.py
@@ -401,7 +401,7 @@ def better_exec(code, context, text = None, realfile = "<code>", pythonexception
         code = better_compile(code, realfile, realfile)
     try:
         exec(code, get_context(), context)
-    except (bb.BBHandledException, bb.parse.SkipRecipe, bb.data_smart.ExpansionError):
+    except (bb.BBHandledException, bb.parse.SkipRecipe, bb.data_smart.ExpansionError, bb.process.ExecutionError):
         # Error already shown so passthrough, no need for traceback
         raise
     except Exception as e:
-- 
2.30.2

Re: [PATCH] utils: Ensure shell function failure in python logging is correct

[Anton Mikanovich]

26.05.2022 12:47, Anton Mikanovich wrote:
> If a python function exec_func() calls a shell task, the logging wasn't working
> correctly in all cases since the exception was turned into a BBHandledException()
> and the logfile piece was lost which is handled at the top task level.
>
> The easiest way to avoid this is to allow the ExecutionError exceptions to
> be raised to a higher level, we don't need the traceback for them.
>
> Backported from Bitbake 2.0 upstream branch:
> https://git.openembedded.org/bitbake/commit/?id=7cae11f558f9ff5fd05ef23b789aaef92fb5a327
>
> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
This is temporary backport until we will migrate to the new Bitbake version
(requires override syntax fix for all the layers).

We are using 1.50 Bitabake branch which is already in 'End of life' state.
Branch 1.52 includes the fix, but will be EOL this month.
So we should migrate to Bitbake 2.0 branch which is LTS and will be 
supported
for the next 2 years minimum.

Re: [PATCH] utils: Ensure shell function failure in python logging is correct

[Florian Bezdeka]

On 26.05.22 11:47, Anton Mikanovich wrote:
> If a python function exec_func() calls a shell task, the logging wasn't working
> correctly in all cases since the exception was turned into a BBHandledException()
> and the logfile piece was lost which is handled at the top task level.
> 
> The easiest way to avoid this is to allow the ExecutionError exceptions to
> be raised to a higher level, we don't need the traceback for them.

Upstream first? Why can't we ask bitbake guys to do the backport and
lift our bitbake version afterwards? Is that already ongoing?

> 
> Backported from Bitbake 2.0 upstream branch:
> https://git.openembedded.org/bitbake/commit/?id=7cae11f558f9ff5fd05ef23b789aaef92fb5a327
> 
> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> ---
>  bitbake/lib/bb/utils.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
> index 2a150fe..74cbc69 100644
> --- a/bitbake/lib/bb/utils.py
> +++ b/bitbake/lib/bb/utils.py
> @@ -401,7 +401,7 @@ def better_exec(code, context, text = None, realfile = "<code>", pythonexception
>          code = better_compile(code, realfile, realfile)
>      try:
>          exec(code, get_context(), context)
> -    except (bb.BBHandledException, bb.parse.SkipRecipe, bb.data_smart.ExpansionError):
> +    except (bb.BBHandledException, bb.parse.SkipRecipe, bb.data_smart.ExpansionError, bb.process.ExecutionError):
>          # Error already shown so passthrough, no need for traceback
>          raise
>      except Exception as e:

Re: [PATCH] utils: Ensure shell function failure in python logging is correct

[Anton Mikanovich]

27.05.2022 10:53, Florian Bezdeka wrote:
> Upstream first? Why can't we ask bitbake guys to do the backport and
> lift our bitbake version afterwards? Is that already ongoing?
Because BItbake branch we use (1.50) is already in 'End of Life' state.
This fix was backported to 1.52 which already use new override syntax.

Re: [PATCH] utils: Ensure shell function failure in python logging is correct

[Baurzhan Ismagulov]

On Fri, May 27, 2022 at 09:53:24AM +0200, Florian Bezdeka wrote:
> On 26.05.22 11:47, Anton Mikanovich wrote:
> > If a python function exec_func() calls a shell task, the logging wasn't working
> > correctly in all cases since the exception was turned into a BBHandledException()
> > and the logfile piece was lost which is handled at the top task level.
> > 
> > The easiest way to avoid this is to allow the ExecutionError exceptions to
> > be raised to a higher level, we don't need the traceback for them.
> 
> Upstream first? Why can't we ask bitbake guys to do the backport and
> lift our bitbake version afterwards? Is that already ongoing?

We had checked https://wiki.yoctoproject.org/wiki/Releases:

* 1.50 is EOL
* 1.52 is untested with Isar and its support ends on Tue

That is why I think the optimal next step is to migrate to bitbake 2.0 after
the upcoming Isar release. This patch addresses the request to get the problem
fixed ASAP before the release. If anyone volunteers to drive the upstream
discussion -- I don't have anything against it, but I'd be reluctant to delay
the Isar release if it takes longer.

With kind regards,
Baurzhan.

Re: [PATCH] utils: Ensure shell function failure in python logging is correct

[Jan Kiszka]

On 27.05.22 10:07, Baurzhan Ismagulov wrote:
> On Fri, May 27, 2022 at 09:53:24AM +0200, Florian Bezdeka wrote:
>> On 26.05.22 11:47, Anton Mikanovich wrote:
>>> If a python function exec_func() calls a shell task, the logging wasn't working
>>> correctly in all cases since the exception was turned into a BBHandledException()
>>> and the logfile piece was lost which is handled at the top task level.
>>>
>>> The easiest way to avoid this is to allow the ExecutionError exceptions to
>>> be raised to a higher level, we don't need the traceback for them.
>>
>> Upstream first? Why can't we ask bitbake guys to do the backport and
>> lift our bitbake version afterwards? Is that already ongoing?
> 
> We had checked https://wiki.yoctoproject.org/wiki/Releases:
> 
> * 1.50 is EOL
> * 1.52 is untested with Isar and its support ends on Tue
> 
> That is why I think the optimal next step is to migrate to bitbake 2.0 after
> the upcoming Isar release. This patch addresses the request to get the problem
> fixed ASAP before the release. If anyone volunteers to drive the upstream
> discussion -- I don't have anything against it, but I'd be reluctant to delay
> the Isar release if it takes longer.
> 

Agreed, we will very likely not have the power to re-active an EOL
branch of bitbake.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: [PATCH] utils: Ensure shell function failure in python logging is correct

[Anton Mikanovich]

26.05.2022 12:47, Anton Mikanovich wrote:
> If a python function exec_func() calls a shell task, the logging wasn't working
> correctly in all cases since the exception was turned into a BBHandledException()
> and the logfile piece was lost which is handled at the top task level.
>
> The easiest way to avoid this is to allow the ExecutionError exceptions to
> be raised to a higher level, we don't need the traceback for them.
>
> Backported from Bitbake 2.0 upstream branch:
> https://git.openembedded.org/bitbake/commit/?id=7cae11f558f9ff5fd05ef23b789aaef92fb5a327
>
> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>

Applied to next.