Re: sbuild race with sporadic build breakages

[Jan Kiszka <jan.kiszka@siemens.com>]

On 17.06.24 10:32, Baurzhan Ismagulov wrote:
> On 2024-06-17 09:44, Jan Kiszka wrote:
>> On 24.05.24 13:55, Anton Mikanovich wrote:
>>> 24/05/2024 13:42, Jan Kiszka wrote:
>>>> Why speaks against this?
>>>>
>>>> diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
>>>> index f1193c20..3f27a2ef 100644
>>>> --- a/meta/classes/sbuild.bbclass
>>>> +++ b/meta/classes/sbuild.bbclass
>>>> @@ -7,7 +7,7 @@ SCHROOT_MOUNTS ?= ""
>>>>     inherit crossvars
>>>>   -SBUILD_CHROOT ?=
>>>> "${DEBDISTRONAME}-${SCHROOT_USER}-${ISAR_BUILD_UUID}-${@os.getpid()}"
>>>> +SBUILD_CHROOT ?=
>>>> "${DEBDISTRONAME}-${SCHROOT_USER}-${ISAR_BUILD_UUID}-${BB_TASKHASH}"
>>>>     SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}"
>>>>   SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}"
>>>>
>>>> We definitely need to get rid of that getpid.
>>>>
>>> Ok, I can prepare a patch and we can try.
>>> The only point we shouldn't loose is task-taskhash mapping, but hope it
>>> will
>>> be ok with the existing logging of schroot session during its creation.
>>
>> What's the status of this? Just triggered this bug once again over isar
>> next, reminding me of it.
> 
> Anton will test and send the patch these days.
> 
> We are still analyzing the root cause. The issue is not reliably reproducible.
> Given that bitbake differentiates tasks by pid in run.do_* and log.do_* files,
> I think the pid itself should be fine. The mechanism of getting the pid could
> have something, although I'm wondering what kind of issues @os.getpid() could
> have.
> 
> Also, Anton's log of 2024-05-24 shows spurious ID change inside a signle sbuild
> call:
> 
> ERROR: Logfile of failure stored in: /workspace/build-auto/isar_master/1364/build/tmp/work/debian-bookworm-mipsel/hello-isar/0.5-b48d156-r0/temp/log.do_dpkg_build.3652871
> ...
> | I: NOTICE: Log filtering will replace 'var/run/schroot/mount/isar-jenkins-6c34c167-bdc8-498f-828a-50cd88b412b0-3652871-9097eeed-6320-4cf9-b2b9-229b1d1fe5a0' with '<<CHROOT>>'
> ...
> | E: /etc/schroot/chroot.d/isar-jenkins-6c34c167-bdc8-498f-828a-50cd88b412b0-2897105: Failed to stat file: No such file or directory
> 
> And this is not a string corruption, this directory did actually exist at some
> point. So looks rather like sbuild having a list of sessions and picking the
> wrong one at some point.
> 
> By saying "We definitely need to get rid of that getpid", what kind of scenario
> do you have in mind?

I think the issue is when the PID is collected into variables. From the
logs, this does not work reliable, therefore the mismatch and the failure.

Jan

-- 
Siemens AG, Technology
Linux Expert Center