isar next getting stuck, most likely on ("REPO_ISAR_DIR") + "/isar.lock"

isar next getting stuck, most likely on ("REPO_ISAR_DIR") + "/isar.lock"

[Henning Schild]

Hi all,

we are seeing isar builds getting stuck when updating from

b78bdb434ef619e13e5c51c74c023fe29c6b3fb6

to
dc36d449dc832db0164f088f3669889557ee015b or beyond.

As of now we did not figure out which patch is causing the issue.

We usually get stuck at some package wanting to do_deploy_deb and then
run into a CI timeout.

2021-09-08 16:56:07 - INFO     - NOTE: recipe
linux-perf-5.10-5.10.26-r0: task do_deploy_deb: Started
2021-09-08 17:43:09 - INFO     - Bitbake still alive (5000s)
ERROR: Job failed: execution took longer than 3h0m0s seconds

One one occasion we logged into that CI runner and simply deleted
("REPO_ISAR_DIR") + "/isar.lock", which got the build un-stuck.

Investigation is ongoing, just to let people know. Maybe others have
seen that as well.

Henning

Re: isar next getting stuck, most likely on ("REPO_ISAR_DIR") + "/isar.lock"

[Anton Mikanovich]

09.09.2021 11:52, Henning Schild wrote:
> Hi all,
>
> we are seeing isar builds getting stuck when updating from
>
> b78bdb434ef619e13e5c51c74c023fe29c6b3fb6
>
> to
> dc36d449dc832db0164f088f3669889557ee015b or beyond.
>
> As of now we did not figure out which patch is causing the issue.
>
> We usually get stuck at some package wanting to do_deploy_deb and then
> run into a CI timeout.
>
> 2021-09-08 16:56:07 - INFO     - NOTE: recipe
> linux-perf-5.10-5.10.26-r0: task do_deploy_deb: Started
> 2021-09-08 17:43:09 - INFO     - Bitbake still alive (5000s)
> ERROR: Job failed: execution took longer than 3h0m0s seconds
>
> One one occasion we logged into that CI runner and simply deleted
> ("REPO_ISAR_DIR") + "/isar.lock", which got the build un-stuck.
>
> Investigation is ongoing, just to let people know. Maybe others have
> seen that as well.
>
> Henning
>
It looks like this information is not actually correct.

After looking into downstream CI I've found at least 15 stuck builds.
Most of them were actually using b78bdb434ef619e13e5c51c74c023fe29c6b3fb6

But there also 3 other hashes used by those stuck builds:
5e563ac4b7072151e6013a8f516c2e346e8677db
dc36d449dc832db0164f088f3669889557ee015b
e274130b870b7e31532fe191b9932cde1d819b4b

The first time it was observed was 2021-08-18 14:33:23 (53199018) with 
Isar on b78bdb4
The last one - 2021-09-10 19:16:49 (54858872) also with Isar on b78bdb4

So it is definitely not something started after switching from b78bdb4 
to dc36d44, but something happened before.

-- 
Anton Mikanovich
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov