Re: sbuild race with sporadic build breakages

[Jan Kiszka <jan.kiszka@siemens.com>]

On 24.05.24 10:08, Anton Mikanovich wrote:
> 23/05/2024 20:08, 'Jan Kiszka' via isar-users wrote:
>> Hi all,
>>
>> I think I've seen it before but didn't look closer so far:
>>
>> 2024-05-23 15:16:22 - INFO     - Log data follows:
>> 2024-05-23 15:16:22 - INFO     - | DEBUG: Executing python function
>> sstate_task_prefunc
>> 2024-05-23 15:16:22 - INFO     - | DEBUG: Python function
>> sstate_task_prefunc finished
>> 2024-05-23 15:16:22 - INFO     - | DEBUG: Executing python function
>> do_dpkg_build
>> 2024-05-23 15:16:22 - INFO     - | DEBUG: Executing shell function
>> dpkg_schroot_create_configs
>> 2024-05-23 15:16:22 - INFO     - | Creating
>> /etc/schroot/chroot.d/isar-builder-3cef593b-e753-4463-a544-cea3d14532e2-39837
>> 2024-05-23 15:16:22 - INFO     - | DEBUG: Shell function
>> dpkg_schroot_create_configs finished
>> 2024-05-23 15:16:22 - INFO     - | DEBUG: Executing shell function
>> dpkg_runbuild
>> 2024-05-23 15:16:22 - INFO     - | sbuild (Debian sbuild) 0.85.0 (04
>> January 2023) on runner-czqurwfpg-project-10191315-concurrent-25-hauy0e5m
>> 2024-05-23 15:16:22 - INFO     - |
>> 2024-05-23 15:16:22 - INFO     - |
>> +==============================================================================+
>> 2024-05-23 15:16:22 - INFO     - | | linux-cip-rt
>> 5.10.214-cip46-rt19+r0 (arm64)  Thu, 23 May 2024 15:16:21 +0000 |
>> 2024-05-23 15:16:22 - INFO     - |
>> +==============================================================================+
>> 2024-05-23 15:16:22 - INFO     - |
>> 2024-05-23 15:16:22 - INFO     - | Package: linux-cip-rt
>> 2024-05-23 15:16:22 - INFO     - | Version: 5.10.214-cip46-rt19+r0
>> 2024-05-23 15:16:22 - INFO     - | Source Version: 5.10.214-cip46-rt19+r0
>> 2024-05-23 15:16:22 - INFO     - | Distribution: isar
>> 2024-05-23 15:16:22 - INFO     - | Machine Architecture: amd64
>> 2024-05-23 15:16:22 - INFO     - | Host Architecture: arm64
>> 2024-05-23 15:16:22 - INFO     - | Build Architecture: amd64
>> 2024-05-23 15:16:22 - INFO     - | Build Profiles: kernel debian
>> nolibcdev cross
>> 2024-05-23 15:16:22 - INFO     - | Build Type: binary
>> 2024-05-23 15:16:22 - INFO     - |
>> 2024-05-23 15:16:22 - INFO     - | E:
>> /etc/schroot/chroot.d/isar-builder-3cef593b-e753-4463-a544-cea3d14532e2-41541: Failed to stat file: No such file or directory
>> 2024-05-23 15:16:22 - INFO     - | E: debconf-set-selections failed
>> 2024-05-23 15:16:22 - INFO     - | Failed to set up chroot
>> 2024-05-23 15:16:22 - INFO     - | E: Error creating chroot session:
>> skipping linux-cip-rt
>> (https://gitlab.com/cip-project/cip-core/isar-cip-core/-/jobs/6923376323)
>>
>> So, we are not finding in dpkg_runbuild the schroot we created just
>> before. Looking at why the the former may look for a different schroot,
>> this jumped at me:
>>
>> SBUILD_CHROOT ?=
>> "${DEBDISTRONAME}-${SCHROOT_USER}-${ISAR_BUILD_UUID}-${@os.getpid()}"
>>
>> We are using some pid of the calling process here, and we assume that
>> this pid is only taken once (or at least over the same process) so that
>> there are no difference in SBUILD_CHROOT down the line. But if you look
>> at the log above, that seems to have happened: the last number is
>> different.
>>
>> I'm not yet understanding why the evaluation may happen multiple times
>> and also why it then only rarely causes this, but I also wonder if we
>> shouldn't better enforce an immediate, single evaluation of os.getpid to
>> avoid that risk.
>>
>> Along this, I wonder why we make all those variables here weak. Is there
>> really a use case for overwriting them? Not in-tree, apparently.
>>
>> Jan
>>
> Hello Jan,
> 
> We've already faced with this floating issue and made some investigations.
> As you mentioned, sbuild is using SBUILD_CHROOT value with PID of calling
> process (which is locked to the task and shouldn't be changed during the
> operation). But in some cases sbuild (which was started with correct
> PID) got
> other incorrect value. It can happen on different steps of the package
> build
> process, even on cleanup.
> The most interesting finding is that this new incorrect PID is not just
> random,
> but it is a PID of other sbuild task executed on the same time in
> parallel. It
> means the issue is not in SBUILD_CHROOT expanding, because variables are
> not sharing between different tasks.
> 

So, the variable may actually also not be evaluated per task but at some
point centrally? Then this is even more broken. Let's just use something
like a task hash here, stable and still unique inside a run.

Jan

-- 
Siemens AG, Technology
Linux Expert Center