From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7372243272758984704 X-Forwarded-Encrypted: i=2; AJvYcCWIQ33K+EcZA3GxZABpyEUO1IcNtk4DHOmgwI75cqEhOrTPCTtC7S4GZ67xcLRA+xcxJn7g9FqzBPOX7I9arLNvV1CMtr02H9mcsPo= X-Received: by 2002:a5d:42cd:0:b0:355:3ee:8fee with SMTP id ffacd0b85a97d-35526c38d0fmr815681f8f.27.1716538086752; Fri, 24 May 2024 01:08:06 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:adf:e683:0:b0:34e:d12d:647 with SMTP id ffacd0b85a97d-35504eeea3cls221584f8f.0.-pod-prod-01-eu; Fri, 24 May 2024 01:08:04 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCXumQKxmF73v5T+8/BhDK/rQ/29oa84jDLEwBxc7x+4MvIC3PNMFysO6JlSqoq35BiLSBItEzgQSjPI7W+/DFGkpDaXoNX8BcG0ob4= X-Google-Smtp-Source: AGHT+IEdKGVLHAoLgYhDnkvVOQTXgh8bbF2HJPBk7pjlLYJg4U9dsWPHh1kE6r5KLGx04xqUKOMI X-Received: by 2002:a5d:5510:0:b0:34d:7f9a:ecbd with SMTP id ffacd0b85a97d-355245e3b3bmr946020f8f.1.1716538084571; Fri, 24 May 2024 01:08:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1716538084; cv=none; d=google.com; s=arc-20160816; b=XIjWsNbpP7KQFbKLL9Hh5fNF6PDDBUgbxeSWQZsXc23ky6fXZYlvRHG6tDFlHL3BtA DcfWD+pb6Q4XAai0Qa77peHllUKzqOe+9oafMwsr2Jd3GjADjoxfsYY7NHLk9aXJCfGf BQ+4SOI1u2wK83XmRzpe77I3iL1OQJQlC6GCZdMbu035/mWIYP62xw4TiHUo7wECUYwG y1TMjF0iH45BsWD/ccvoRBrwPB8DqLCbX5j0l6ekB+DsmwC8khcKe5JDa4lhg4XApXQb XLByGbUGe28SH5zmO7zMAi0+wO/NgmPitlRx1/Po1MYj8bAmI0hpSW117a8GOD17ej+n 1Ruw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:in-reply-to:from:content-language :references:to:subject:user-agent:mime-version:date:message-id; bh=VxtsxRs7XpyKdDbr/ldRYK9zBkoiUVV0dJIi663KAXg=; fh=WihjhDXEbM6E1y1glmk2nHMgMd+1ZI4Af8ke6D9d+1w=; b=cbaLxEwjtDwJCodbxMyR3RHzkRCzD0wVMHwGeB9BEpyGfblyA1cwg4y66JM0uqxSPL xc0rkPYHmiqrLC4WZHMVhn7NjBmEZZngXPB6HUTqrTF30UBDtiR+4WHA3Qj2Ounv3lPC 8FcpRbQlAhhnULoRRu3LVvaL1OnslEs4rV5PPTTf8kxb2TimhdRTxf6QpsgXNlFhBttv Lb2r/Y1RreUwzj+N7ESTtKkgJO04YHRoURPVI/kdGCMksqMP2eT4QilhIoMz3Y80kPOr oQTsruarEXuYSXFuMPPVU2GBNlIgkyWc4uQPu0JUriAqqwBQcjBWDd4KeyUROun1xlYs j+Vw==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id ffacd0b85a97d-35559d53ff6si14427f8f.0.2024.05.24.01.08.04 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Fri, 24 May 2024 01:08:04 -0700 (PDT) Received-SPF: pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Received: from [127.0.0.1] (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 44O881tN018113 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 May 2024 10:08:03 +0200 Message-ID: Date: Fri, 24 May 2024 11:08:01 +0300 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: sbuild race with sporadic build breakages To: Jan Kiszka , isar-users , Uladzimir Bely References: <187215a1-ff1b-48b4-979f-9b49689a78e7@siemens.com> Content-Language: en-US, ru-RU From: Anton Mikanovich In-Reply-To: <187215a1-ff1b-48b4-979f-9b49689a78e7@siemens.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: xS5jjpE+tZjx 23/05/2024 20:08, 'Jan Kiszka' via isar-users wrote: > Hi all, > > I think I've seen it before but didn't look closer so far: > > 2024-05-23 15:16:22 - INFO - Log data follows: > 2024-05-23 15:16:22 - INFO - | DEBUG: Executing python function sstate_task_prefunc > 2024-05-23 15:16:22 - INFO - | DEBUG: Python function sstate_task_prefunc finished > 2024-05-23 15:16:22 - INFO - | DEBUG: Executing python function do_dpkg_build > 2024-05-23 15:16:22 - INFO - | DEBUG: Executing shell function dpkg_schroot_create_configs > 2024-05-23 15:16:22 - INFO - | Creating /etc/schroot/chroot.d/isar-builder-3cef593b-e753-4463-a544-cea3d14532e2-39837 > 2024-05-23 15:16:22 - INFO - | DEBUG: Shell function dpkg_schroot_create_configs finished > 2024-05-23 15:16:22 - INFO - | DEBUG: Executing shell function dpkg_runbuild > 2024-05-23 15:16:22 - INFO - | sbuild (Debian sbuild) 0.85.0 (04 January 2023) on runner-czqurwfpg-project-10191315-concurrent-25-hauy0e5m > 2024-05-23 15:16:22 - INFO - | > 2024-05-23 15:16:22 - INFO - | +==============================================================================+ > 2024-05-23 15:16:22 - INFO - | | linux-cip-rt 5.10.214-cip46-rt19+r0 (arm64) Thu, 23 May 2024 15:16:21 +0000 | > 2024-05-23 15:16:22 - INFO - | +==============================================================================+ > 2024-05-23 15:16:22 - INFO - | > 2024-05-23 15:16:22 - INFO - | Package: linux-cip-rt > 2024-05-23 15:16:22 - INFO - | Version: 5.10.214-cip46-rt19+r0 > 2024-05-23 15:16:22 - INFO - | Source Version: 5.10.214-cip46-rt19+r0 > 2024-05-23 15:16:22 - INFO - | Distribution: isar > 2024-05-23 15:16:22 - INFO - | Machine Architecture: amd64 > 2024-05-23 15:16:22 - INFO - | Host Architecture: arm64 > 2024-05-23 15:16:22 - INFO - | Build Architecture: amd64 > 2024-05-23 15:16:22 - INFO - | Build Profiles: kernel debian nolibcdev cross > 2024-05-23 15:16:22 - INFO - | Build Type: binary > 2024-05-23 15:16:22 - INFO - | > 2024-05-23 15:16:22 - INFO - | E: /etc/schroot/chroot.d/isar-builder-3cef593b-e753-4463-a544-cea3d14532e2-41541: Failed to stat file: No such file or directory > 2024-05-23 15:16:22 - INFO - | E: debconf-set-selections failed > 2024-05-23 15:16:22 - INFO - | Failed to set up chroot > 2024-05-23 15:16:22 - INFO - | E: Error creating chroot session: skipping linux-cip-rt > (https://gitlab.com/cip-project/cip-core/isar-cip-core/-/jobs/6923376323) > > So, we are not finding in dpkg_runbuild the schroot we created just > before. Looking at why the the former may look for a different schroot, > this jumped at me: > > SBUILD_CHROOT ?= "${DEBDISTRONAME}-${SCHROOT_USER}-${ISAR_BUILD_UUID}-${@os.getpid()}" > > We are using some pid of the calling process here, and we assume that > this pid is only taken once (or at least over the same process) so that > there are no difference in SBUILD_CHROOT down the line. But if you look > at the log above, that seems to have happened: the last number is > different. > > I'm not yet understanding why the evaluation may happen multiple times > and also why it then only rarely causes this, but I also wonder if we > shouldn't better enforce an immediate, single evaluation of os.getpid to > avoid that risk. > > Along this, I wonder why we make all those variables here weak. Is there > really a use case for overwriting them? Not in-tree, apparently. > > Jan > Hello Jan, We've already faced with this floating issue and made some investigations. As you mentioned, sbuild is using SBUILD_CHROOT value with PID of calling process (which is locked to the task and shouldn't be changed during the operation). But in some cases sbuild (which was started with correct PID) got other incorrect value. It can happen on different steps of the package build process, even on cleanup. The most interesting finding is that this new incorrect PID is not just random, but it is a PID of other sbuild task executed on the same time in parallel. It means the issue is not in SBUILD_CHROOT expanding, because variables are not sharing between different tasks.