Re: sbuild race with sporadic build breakages

["MOESSBAUER, Felix" <felix.moessbauer@siemens.com>]

On Thu, 2024-05-23 at 19:08 +0200, 'Jan Kiszka' via isar-users wrote:
> Hi all,
> 
> I think I've seen it before but didn't look closer so far:
> 
> 2024-05-23 15:16:22 - INFO     - Log data follows:
> 2024-05-23 15:16:22 - INFO     - | DEBUG: Executing python function
> sstate_task_prefunc
> 2024-05-23 15:16:22 - INFO     - | DEBUG: Python function
> sstate_task_prefunc finished
> 2024-05-23 15:16:22 - INFO     - | DEBUG: Executing python function
> do_dpkg_build
> 2024-05-23 15:16:22 - INFO     - | DEBUG: Executing shell function
> dpkg_schroot_create_configs
> 2024-05-23 15:16:22 - INFO     - | Creating
> /etc/schroot/chroot.d/isar-builder-3cef593b-e753-4463-a544-
> cea3d14532e2-39837
> 2024-05-23 15:16:22 - INFO     - | DEBUG: Shell function
> dpkg_schroot_create_configs finished
> 2024-05-23 15:16:22 - INFO     - | DEBUG: Executing shell function
> dpkg_runbuild
> 2024-05-23 15:16:22 - INFO     - | sbuild (Debian sbuild) 0.85.0 (04
> January 2023) on runner-czqurwfpg-project-10191315-concurrent-25-
> hauy0e5m
> 2024-05-23 15:16:22 - INFO     - |
> 2024-05-23 15:16:22 - INFO     - |
> +====================================================================
> ==========+
> 2024-05-23 15:16:22 - INFO     - | | linux-cip-rt 5.10.214-cip46-
> rt19+r0 (arm64)  Thu, 23 May 2024 15:16:21 +0000 |
> 2024-05-23 15:16:22 - INFO     - |
> +====================================================================
> ==========+
> 2024-05-23 15:16:22 - INFO     - |
> 2024-05-23 15:16:22 - INFO     - | Package: linux-cip-rt
> 2024-05-23 15:16:22 - INFO     - | Version: 5.10.214-cip46-rt19+r0
> 2024-05-23 15:16:22 - INFO     - | Source Version: 5.10.214-cip46-
> rt19+r0
> 2024-05-23 15:16:22 - INFO     - | Distribution: isar
> 2024-05-23 15:16:22 - INFO     - | Machine Architecture: amd64
> 2024-05-23 15:16:22 - INFO     - | Host Architecture: arm64
> 2024-05-23 15:16:22 - INFO     - | Build Architecture: amd64
> 2024-05-23 15:16:22 - INFO     - | Build Profiles: kernel debian
> nolibcdev cross
> 2024-05-23 15:16:22 - INFO     - | Build Type: binary
> 2024-05-23 15:16:22 - INFO     - |
> 2024-05-23 15:16:22 - INFO     - | E: /etc/schroot/chroot.d/isar-
> builder-3cef593b-e753-4463-a544-cea3d14532e2-41541: Failed to stat
> file: No such file or directory
> 2024-05-23 15:16:22 - INFO     - | E: debconf-set-selections failed
> 2024-05-23 15:16:22 - INFO     - | Failed to set up chroot
> 2024-05-23 15:16:22 - INFO     - | E: Error creating chroot session:
> skipping linux-cip-rt
> (
> https://gitlab.com/cip-project/cip-core/isar-cip-core/-/jobs/692337632
> 3)
> 
> So, we are not finding in dpkg_runbuild the schroot we created just 
> before. Looking at why the the former may look for a different
> schroot, 
> this jumped at me:
> 
> SBUILD_CHROOT ?=
> "${DEBDISTRONAME}-${SCHROOT_USER}-${ISAR_BUILD_UUID}-${@os.getpid()}"
> 
> We are using some pid of the calling process here, and we assume that
> this pid is only taken once (or at least over the same process) so
> that 
> there are no difference in SBUILD_CHROOT down the line. But if you
> look 
> at the log above, that seems to have happened: the last number is 
> different.
> 
> I'm not yet understanding why the evaluation may happen multiple
> times 
> and also why it then only rarely causes this, but I also wonder if we
> shouldn't better enforce an immediate, single evaluation of os.getpid
> to 
> avoid that risk.

During parsing, variables are evaluated multiple times, e.g. because
they happen in different branches of the parsing operation. The problem
here might be the lazy expansion of the variable, though. By that, the
SBUILD_CHROOT variable is expanded when the final value is needed (e.g.
via a d.getVar()). If the SBUILD_CHROOT is used in various context, the
expansion also happens multiple times. We should better use immediate
expansion here (at least for the getpid part).

CC'ing Adriaan as he is more familiar with the parser.

Felix

> 
> Along this, I wonder why we make all those variables here weak. Is
> there 
> really a use case for overwriting them? Not in-tree, apparently.
> 
> Jan
> 
> -- 
> Siemens AG, Technology
> Linux Expert Center
> 

-- 
Siemens AG, Technology
Linux Expert Center