From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7372243272758984704 X-Forwarded-Encrypted: i=2; AJvYcCXMBbSiocpyyM9HZQ1ZXJmanbpFtBdeUQ5eVq8my0S6lS1QA/i8MFKtFTtF/QJWGetfrEOEROKAyZt+lNtY7mT+FG1ZHlCuM0MMc78= X-Received: by 2002:a5d:548b:0:b0:35f:1237:cd00 with SMTP id ffacd0b85a97d-366e94d95c3mr2198807f8f.41.1719216019853; Mon, 24 Jun 2024 01:00:19 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6000:101:b0:35f:1b61:e5e1 with SMTP id ffacd0b85a97d-36480e1c2ccls1474512f8f.1.-pod-prod-07-eu; Mon, 24 Jun 2024 01:00:16 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCXQZ5b/DD/9k/V2dc53NUSdPsLcfEAiD7bFKkyjhq0SXEYenBxvXV3ESezPgl66tjATIlumlMXqrRE4HqMcUmwVHHIVMamfIKBlfmc= X-Google-Smtp-Source: AGHT+IFoeieyaPtsjepEQiVhS6ZcoV8OMYv2Dah9zRRXwoHKezSWhxEiMxtwzYYMqKYSWWNpSwO1 X-Received: by 2002:a05:6000:45a0:b0:362:ffb4:30b8 with SMTP id ffacd0b85a97d-366e94d99a3mr2168641f8f.42.1719216016262; Mon, 24 Jun 2024 01:00:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1719216016; cv=none; d=google.com; s=arc-20160816; b=Jhg7E4hVNifRAEXupGGxWw/7Agg/qUbFBNl5tVfRf9IsrtE2O75Pr5nXmO6spMecds 2hhhUCyBiV/kbgyKwvi1vwf8WmUa+igm5fL6snJL6TiD6iFO8j1dVd24UKWiNNlj+Wo0 DnGdTG+Q0vgafVPSrTYy3QhTQwsjpwr3AIWYieC4RVRQriDZoakqUKtoXB9K367syNdH Ap6b2VOVC7j/r9ie9MDRH4TOWSA5O2jybtng78BF7bvteb3XGefdXBAO6FDhWXt1iPnd KWLcI7UDD1oKT+84ivgwjBbwZFuAwJcRrLM3AC+biM5kDYUYVX62cVudS0/mTghSCRSK vr8g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:cc:to:from:subject:message-id; bh=11lBfIBFEOhuMbqnB+BqVqcTAOf3V6Pz+aKp9QujVAo=; fh=WGLAO9cTPbYyguaCYimanFse/ngsm1yXA2j9sU/CZEM=; b=U8FSJhfna1G3BdkbvnvESgFioNv3VlKxc9aKOT5FnjYKkj0WD9a+xjkVDaIMBw0FQP Pmy5fiGRzG3n2VFFH6wCYQmRxXWmOb1LCUf1UkxlmB52XtHDIYNKlb068G5gP5YjJvri t3IBtyCsWNmOAR/o3N1Tbpzmhxwa+qJMauPIv+E0il4jRoRq0/8uTB/NfQgXMmwxI8if ZKserYQhzF5VaqgSuIUBZtuv4lNgrkxc7RZclEsN8dIvdBRCV/KTuY+aGzWwo1SOYW1T UFGTFipakrcDksEqpsKq5zWzVRLtwvNZMhXDZjtakQMtuBK6BZ5As4/gYdzTPLss7hN3 jOXA==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id ffacd0b85a97d-36639f0de49si147768f8f.4.2024.06.24.01.00.15 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 24 Jun 2024 01:00:15 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from [127.0.0.1] (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 45O80B8p019189 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jun 2024 10:00:12 +0200 Message-ID: Subject: Re: sbuild race with sporadic build breakages From: Uladzimir Bely To: Jan Kiszka , isar-users Cc: Felix MOESSBAUER Date: Mon, 24 Jun 2024 11:00:11 +0300 In-Reply-To: <187215a1-ff1b-48b4-979f-9b49689a78e7@siemens.com> References: <187215a1-ff1b-48b4-979f-9b49689a78e7@siemens.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.52.2 (by Flathub.org) MIME-Version: 1.0 X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: O3/RR+e1opCy We were running CI with the patch from https://groups.google.com/g/isar-users/c/koNMYOyo-YM applied for several days and at some moment the issue happened again (with "heavy" NoCross test): ``` Creating /etc/schroot/chroot.d/isar-jenkins-7a841d41-9632-420f-a97d- bbf30e2732b9- 9f8d1f2d22de72cfbecc8e6ed8ceb6f59a829def67a2b9b7e2eb45a82af88ef2 DEBUG: Shell function dpkg_schroot_create_configs finished DEBUG: Executing shell function dpkg_runbuild sbuild (Debian sbuild) 0.85.0 (04 January 2023) on turar.m.ilbers.de +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D+ | libhello 0.3 (mipsel) Sat, 22 Jun 2024 12:34:22 +0000 | +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D+ Package: libhello Version: 0.3 Source Version: 0.3 Distribution: isar Machine Architecture: amd64 Host Architecture: mipsel Build Architecture: mipsel Build Type: binary I: NOTICE: Log filtering will replace 'var/run/schroot/mount/isar- jenkins-7a841d41-9632-420f-a97d-bbf30e2732b9- 9f8d1f2d22de72cfbecc8e6ed8ceb6f59a829def67a2b9b7e2eb45a82af88ef2- 3ab2cd48-c709-4e9e-888f-a45fb2da713e' with '<>' +---------------------------------------------------------------------- --------+ | Chroot Setup Commands =20 | +---------------------------------------------------------------------- --------+ echo "Package: *\nPin: release n=3Disar\nPin-Priority: 1000" > /etc/apt/preferences.d/isar-apt ----------------------------------------------------------------------- --------------------- I: Finished running 'echo "Package: *\nPin: release n=3Disar\nPin- Priority: 1000" > /etc/apt/preferences.d/isar-apt'. echo "APT::Get::allow-downgrades 1;" > /etc/apt/apt.conf.d/50isar-apt --------------------------------------------------------------------- I: Finished running 'echo "APT::Get::allow-downgrades 1;" > /etc/apt/apt.conf.d/50isar-apt'. rm -f /var/log/dpkg.log ----------------------- I: Finished running 'rm -f /var/log/dpkg.log'. mkdir -p /var/cache/apt/archives -------------------------------- I: Finished running 'mkdir -p /var/cache/apt/archives'. find /home/builder/libhello/rootfs/var/cache/apt/archives -maxdepth 1 - name '*.deb' -exec ln -t /var/cache/apt/archives/ -sf {} + ----------------------------------------------------------------------- ---------------------------------------------------------- I: Finished running 'find /home/builder/libhello/rootfs/var/cache/apt/archives -maxdepth 1 -name '*.deb' -exec ln -t /var/cache/apt/archives/ -sf {} +'. apt-get update -o Dir::Etc::SourceList=3D"sources.list.d/isar-apt.list" - o Dir::Etc::SourceParts=3D"-" -o APT::Get::List-Cleanup=3D"0" ----------------------------------------------------------------------- --------------------------------------------------------- Get:1 file:/isar-apt isar InRelease Ign:1 file:/isar-apt isar InRelease Get:2 file:/isar-apt isar Release [5006 B] Get:2 file:/isar-apt isar Release [5006 B] Get:3 file:/isar-apt isar Release.gpg Ign:3 file:/isar-apt isar Release.gpg Reading package lists... I: Finished running 'apt-get update -o Dir::Etc::SourceList=3D"sources.list.d/isar-apt.list" -o Dir::Etc::SourceParts=3D"-" -o APT::Get::List-Cleanup=3D"0"'. Finished processing commands. ----------------------------------------------------------------------- --------- I: NOTICE: Log filtering will replace 'build/libhello-BBthl5/resolver- X1hxT7' with '<>' +---------------------------------------------------------------------- --------+ | Update chroot =20 | +---------------------------------------------------------------------- --------+ Reading package lists... Building dependency tree... Reading state information... Calculating upgrade... 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. +---------------------------------------------------------------------- --------+ | Fetch source files =20 | +---------------------------------------------------------------------- --------+ Local sources ------------- /workspace/build/isar_ub_devel/436/build/tmp/work/debian-bookworm- mipsel/libhello/0.3-f4a5669-r0/libhello_0.3.dsc exists in /workspace/build/isar_ub_devel/436/build/tmp/work/debian-bookworm- mipsel/libhello/0.3-f4a5669-r0; copying to chroot I: NOTICE: Log filtering will replace 'build/libhello-BBthl5/libhello- 0.3' with '<>' I: NOTICE: Log filtering will replace 'build/libhello-BBthl5' with '<>' +---------------------------------------------------------------------- --------+ | Install package build dependencies =20 | +---------------------------------------------------------------------- --------+ Setup apt archive ----------------- E: /var/lib/schroot/session/isar-jenkins-7a841d41-9632-420f-a97d- bbf30e2732b9- 8641cd44af3718c57bd0c9e341b9dd285c91d116b7bb1d5b93a87a4e4c043d01- 0f576f86-4a3b-4d43-a366-f14782e97097: Failed to open file: No such file or directory E: Can't chmod /<> to 0770: No such file or directory E: Failed to set 0770 permissions on dummy package dir E: Setting up apt archive failed ``` And that's what we see in logs: ``` build/tmp$ grep -R f14782e97097 work/debian-bookworm-mipsel/libhello/0.3-f4a5669- r0/temp/log.do_dpkg_build:E: /var/lib/schroot/session/isar-jenkins- 7a841d41-9632-420f-a97d-bbf30e2732b9- 8641cd44af3718c57bd0c9e341b9dd285c91d116b7bb1d5b93a87a4e4c043d01- 0f576f86-4a3b-4d43-a366-f14782e97097: Failed to open file: No such file or directory work/debian-bookworm-mipsel/libhello/0.3-f4a5669- r0/temp/log.do_dpkg_build.694689:E: /var/lib/schroot/session/isar- jenkins-7a841d41-9632-420f-a97d-bbf30e2732b9- 8641cd44af3718c57bd0c9e341b9dd285c91d116b7bb1d5b93a87a4e4c043d01- 0f576f86-4a3b-4d43-a366-f14782e97097: Failed to open file: No such file or directory work/debian-bookworm-amd64/kselftest/5.6.13- r0/temp/log.do_dpkg_build.369632:I: NOTICE: Log filtering will replace 'var/run/schroot/mount/isar-jenkins-7a841d41-9632-420f-a97d- bbf30e2732b9- 8641cd44af3718c57bd0c9e341b9dd285c91d116b7bb1d5b93a87a4e4c043d01- 0f576f86-4a3b-4d43-a366-f14782e97097' with '<>' work/debian-bookworm-amd64/kselftest/5.6.13- r0/temp/log.do_dpkg_build.369632:SCHROOT_SESSION_ID=3Disar-jenkins- 7a841d41-9632-420f-a97d-bbf30e2732b9- 8641cd44af3718c57bd0c9e341b9dd285c91d116b7bb1d5b93a87a4e4c043d01- 0f576f86-4a3b-4d43-a366-f14782e97097 ``` For some reason, when building "libhello" package for "mipsel" architecture, sbuild tried to use existing directory, that "kselftest" package form "amd64" used. So, replacing pid with task hashsum does not help with this sporadic issue (but reduces readability of log files). We'll continue debugging the root cause of the issue. --=20 Best regards, Uladzimir.