Re: sbuild race with sporadic build breakages

[Uladzimir Bely <ubely@ilbers.de>]

We were running CI with the patch from
https://groups.google.com/g/isar-users/c/koNMYOyo-YM applied for
several days and at some moment the issue happened again (with "heavy"
NoCross test):

```
Creating /etc/schroot/chroot.d/isar-jenkins-7a841d41-9632-420f-a97d-
bbf30e2732b9-
9f8d1f2d22de72cfbecc8e6ed8ceb6f59a829def67a2b9b7e2eb45a82af88ef2
DEBUG: Shell function dpkg_schroot_create_configs finished
DEBUG: Executing shell function dpkg_runbuild
sbuild (Debian sbuild) 0.85.0 (04 January 2023) on turar.m.ilbers.de

+======================================================================
========+
| libhello 0.3 (mipsel)                        Sat, 22 Jun 2024
12:34:22 +0000 |
+======================================================================
========+

Package: libhello
Version: 0.3
Source Version: 0.3
Distribution: isar
Machine Architecture: amd64
Host Architecture: mipsel
Build Architecture: mipsel
Build Type: binary

I: NOTICE: Log filtering will replace 'var/run/schroot/mount/isar-
jenkins-7a841d41-9632-420f-a97d-bbf30e2732b9-
9f8d1f2d22de72cfbecc8e6ed8ceb6f59a829def67a2b9b7e2eb45a82af88ef2-
3ab2cd48-c709-4e9e-888f-a45fb2da713e' with '<<CHROOT>>'

+----------------------------------------------------------------------
--------+
| Chroot Setup Commands                                               
|
+----------------------------------------------------------------------
--------+


echo "Package: *\nPin: release n=isar\nPin-Priority: 1000" >
/etc/apt/preferences.d/isar-apt
-----------------------------------------------------------------------
---------------------


I: Finished running 'echo "Package: *\nPin: release n=isar\nPin-
Priority: 1000" > /etc/apt/preferences.d/isar-apt'.

echo "APT::Get::allow-downgrades 1;" > /etc/apt/apt.conf.d/50isar-apt
---------------------------------------------------------------------


I: Finished running 'echo "APT::Get::allow-downgrades 1;" >
/etc/apt/apt.conf.d/50isar-apt'.

rm -f /var/log/dpkg.log
-----------------------


I: Finished running 'rm -f /var/log/dpkg.log'.

mkdir -p /var/cache/apt/archives
--------------------------------


I: Finished running 'mkdir -p /var/cache/apt/archives'.

find /home/builder/libhello/rootfs/var/cache/apt/archives -maxdepth 1 -
name '*.deb' -exec ln -t /var/cache/apt/archives/ -sf {} +
-----------------------------------------------------------------------
----------------------------------------------------------


I: Finished running 'find
/home/builder/libhello/rootfs/var/cache/apt/archives -maxdepth 1 -name
'*.deb' -exec ln -t /var/cache/apt/archives/ -sf {} +'.

apt-get update -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" -
o Dir::Etc::SourceParts="-" -o APT::Get::List-Cleanup="0"
-----------------------------------------------------------------------
---------------------------------------------------------

Get:1 file:/isar-apt isar InRelease
Ign:1 file:/isar-apt isar InRelease
Get:2 file:/isar-apt isar Release [5006 B]
Get:2 file:/isar-apt isar Release [5006 B]
Get:3 file:/isar-apt isar Release.gpg
Ign:3 file:/isar-apt isar Release.gpg
Reading package lists...

I: Finished running 'apt-get update -o
Dir::Etc::SourceList="sources.list.d/isar-apt.list" -o
Dir::Etc::SourceParts="-" -o APT::Get::List-Cleanup="0"'.

Finished processing commands.
-----------------------------------------------------------------------
---------
I: NOTICE: Log filtering will replace 'build/libhello-BBthl5/resolver-
X1hxT7' with '<<RESOLVERDIR>>'

+----------------------------------------------------------------------
--------+
| Update chroot                                                       
|
+----------------------------------------------------------------------
--------+

Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

+----------------------------------------------------------------------
--------+
| Fetch source files                                                  
|
+----------------------------------------------------------------------
--------+


Local sources
-------------

/workspace/build/isar_ub_devel/436/build/tmp/work/debian-bookworm-
mipsel/libhello/0.3-f4a5669-r0/libhello_0.3.dsc exists in
/workspace/build/isar_ub_devel/436/build/tmp/work/debian-bookworm-
mipsel/libhello/0.3-f4a5669-r0; copying to chroot
I: NOTICE: Log filtering will replace 'build/libhello-BBthl5/libhello-
0.3' with '<<PKGBUILDDIR>>'
I: NOTICE: Log filtering will replace 'build/libhello-BBthl5' with
'<<BUILDDIR>>'

+----------------------------------------------------------------------
--------+
| Install package build dependencies                                  
|
+----------------------------------------------------------------------
--------+


Setup apt archive
-----------------

E: /var/lib/schroot/session/isar-jenkins-7a841d41-9632-420f-a97d-
bbf30e2732b9-
8641cd44af3718c57bd0c9e341b9dd285c91d116b7bb1d5b93a87a4e4c043d01-
0f576f86-4a3b-4d43-a366-f14782e97097: Failed to open file: No such file
or directory
E: Can't chmod /<<RESOLVERDIR>> to 0770: No such file or directory
E: Failed to set 0770 permissions on dummy package dir
E: Setting up apt archive failed
```

And that's what we see in logs:

```
build/tmp$ grep -R f14782e97097
work/debian-bookworm-mipsel/libhello/0.3-f4a5669-
r0/temp/log.do_dpkg_build:E: /var/lib/schroot/session/isar-jenkins-
7a841d41-9632-420f-a97d-bbf30e2732b9-
8641cd44af3718c57bd0c9e341b9dd285c91d116b7bb1d5b93a87a4e4c043d01-
0f576f86-4a3b-4d43-a366-f14782e97097: Failed to open file: No such file
or directory
work/debian-bookworm-mipsel/libhello/0.3-f4a5669-
r0/temp/log.do_dpkg_build.694689:E: /var/lib/schroot/session/isar-
jenkins-7a841d41-9632-420f-a97d-bbf30e2732b9-
8641cd44af3718c57bd0c9e341b9dd285c91d116b7bb1d5b93a87a4e4c043d01-
0f576f86-4a3b-4d43-a366-f14782e97097: Failed to open file: No such file
or directory
work/debian-bookworm-amd64/kselftest/5.6.13-
r0/temp/log.do_dpkg_build.369632:I: NOTICE: Log filtering will replace
'var/run/schroot/mount/isar-jenkins-7a841d41-9632-420f-a97d-
bbf30e2732b9-
8641cd44af3718c57bd0c9e341b9dd285c91d116b7bb1d5b93a87a4e4c043d01-
0f576f86-4a3b-4d43-a366-f14782e97097' with '<<CHROOT>>'
work/debian-bookworm-amd64/kselftest/5.6.13-
r0/temp/log.do_dpkg_build.369632:SCHROOT_SESSION_ID=isar-jenkins-
7a841d41-9632-420f-a97d-bbf30e2732b9-
8641cd44af3718c57bd0c9e341b9dd285c91d116b7bb1d5b93a87a4e4c043d01-
0f576f86-4a3b-4d43-a366-f14782e97097
```

For some reason, when building "libhello" package for "mipsel"
architecture, sbuild tried to use existing directory, that "kselftest"
package form "amd64" used.

So, replacing pid with task hashsum does not help with this sporadic
issue (but reduces readability of log files).

We'll continue debugging the root cause of the issue.

-- 
Best regards,
Uladzimir.