Re: sbuild race with sporadic build breakages

[Anton Mikanovich <amikan@ilbers.de>]

24/05/2024 13:42, Jan Kiszka wrote:
> Why speaks against this?
>
> diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
> index f1193c20..3f27a2ef 100644
> --- a/meta/classes/sbuild.bbclass
> +++ b/meta/classes/sbuild.bbclass
> @@ -7,7 +7,7 @@ SCHROOT_MOUNTS ?= ""
>   
>   inherit crossvars
>   
> -SBUILD_CHROOT ?= "${DEBDISTRONAME}-${SCHROOT_USER}-${ISAR_BUILD_UUID}-${@os.getpid()}"
> +SBUILD_CHROOT ?= "${DEBDISTRONAME}-${SCHROOT_USER}-${ISAR_BUILD_UUID}-${BB_TASKHASH}"
>   
>   SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}"
>   SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}"
>
>
> We definitely need to get rid of that getpid.
>
> Jan
>
Ok, I can prepare a patch and we can try.
The only point we shouldn't loose is task-taskhash mapping, but hope it will
be ok with the existing logging of schroot session during its creation.

Here is one of the logs we've caught on our CI.
The first log output of session ID (with correct PID) is done by Isar.
The second one (also correct) is done by sbuild process later.
And the third (incorrect) log output is done also by the same sbuild process
when the issue happened.
Log:

ERROR: Logfile of failure stored in: 
/workspace/build-auto/isar_master/1364/build/tmp/work/debian-bookworm-mipsel/hello-isar/0.5-b48d156-r0/temp/log.do_dpkg_build.3652871
ERROR: Task 
(mc:qemumipsel-bookworm:/workspace/build-auto/isar_master/1364/meta-isar/recipes-app/hello-isar/hello-isar.bb:do_dpkg_build) 
failed with exit code '1'
| DEBUG: Executing python function sstate_task_prefunc
| DEBUG: Python function sstate_task_prefunc finished
| DEBUG: Executing python function do_dpkg_build
| DEBUG: Executing shell function dpkg_schroot_create_configs
| Creating 
/etc/schroot/chroot.d/isar-jenkins-6c34c167-bdc8-498f-828a-50cd88b412b0-3652871
| DEBUG: Shell function dpkg_schroot_create_configs finished
| DEBUG: Executing shell function dpkg_runbuild
| sbuild (Debian sbuild) 0.85.0 (04 January 2023) on qaton.m.ilbers.de
|
| 
+==============================================================================+
| | hello-isar 0.5 (mipsel)                      Wed, 15 May 2024 
04:12:02 +0000 |
| 
+==============================================================================+
|
| Package: hello-isar
| Version: 0.5
| Source Version: 0.5
| Distribution: isar
| Machine Architecture: amd64
| Host Architecture: mipsel
| Build Architecture: mipsel
| Build Type: binary
|
| I: NOTICE: Log filtering will replace 
'var/run/schroot/mount/isar-jenkins-6c34c167-bdc8-498f-828a-50cd88b412b0-3652871-9097eeed-6320-4cf9-b2b9-229b1d1fe5a0' 
with '<<CHROOT>>'
|
| 
+------------------------------------------------------------------------------+
| | Chroot Setup 
Commands                                                        |
| 
+------------------------------------------------------------------------------+
|
|
| echo "Package: *\nPin: release n=isar\nPin-Priority: 1000" > 
/etc/apt/preferences.d/isar-apt
| 
--------------------------------------------------------------------------------------------
|
|
| I: Finished running 'echo "Package: *\nPin: release 
n=isar\nPin-Priority: 1000" > /etc/apt/preferences.d/isar-apt'.
|
| echo "APT::Get::allow-downgrades 1;" > /etc/apt/apt.conf.d/50isar-apt
| ---------------------------------------------------------------------
|
|
| I: Finished running 'echo "APT::Get::allow-downgrades 1;" > 
/etc/apt/apt.conf.d/50isar-apt'.
|
| rm -f /var/log/dpkg.log
| -----------------------
|
|
| I: Finished running 'rm -f /var/log/dpkg.log'.
|
| mkdir -p /var/cache/apt/archives
| --------------------------------
|
|
| I: Finished running 'mkdir -p /var/cache/apt/archives'.
|
| find /home/builder/hello-isar/rootfs/var/cache/apt/archives -maxdepth 
1 -name '*.deb' -exec ln -t /var/cache/apt/archives/ -sf {} +
| 
-----------------------------------------------------------------------------------------------------------------------------------
|
|
| I: Finished running 'find 
/home/builder/hello-isar/rootfs/var/cache/apt/archives -maxdepth 1 -name 
'*.deb' -exec ln -t /var/cache/apt/archives/ -sf {} +'.
|
| apt-get update -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" 
-o Dir::Etc::SourceParts="-" -o APT::Get::List-Cleanup="0"
| 
--------------------------------------------------------------------------------------------------------------------------------
|
| Get:1 file:/isar-apt isar InRelease
| Ign:1 file:/isar-apt isar InRelease
| Get:2 file:/isar-apt isar Release [5102 B]
| Get:2 file:/isar-apt isar Release [5102 B]
| Get:3 file:/isar-apt isar Release.gpg
| Ign:3 file:/isar-apt isar Release.gpg
| Get:4 file:/isar-apt isar/main Sources [3759 B]
| Get:5 file:/isar-apt isar/main mipsel Packages [3187 B]
| Reading package lists...
|
| I: Finished running 'apt-get update -o 
Dir::Etc::SourceList="sources.list.d/isar-apt.list" -o 
Dir::Etc::SourceParts="-" -o APT::Get::List-Cleanup="0"'.
|
| Finished processing commands.
| 
--------------------------------------------------------------------------------
| I: NOTICE: Log filtering will replace 
'build/hello-isar-gmVcne/resolver-6OF4Lt' with '<<RESOLVERDIR>>'
|
| 
+------------------------------------------------------------------------------+
| | Update chroot |
| 
+------------------------------------------------------------------------------+
|
| Reading package lists...
| Building dependency tree...
| Reading state information...
| Calculating upgrade...
| 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
|
| 
+------------------------------------------------------------------------------+
| | Fetch source 
files                                                           |
| 
+------------------------------------------------------------------------------+
|
|
| Local sources
| -------------
|
| 
/workspace/build-auto/isar_master/1364/build/tmp/work/debian-bookworm-mipsel/hello-isar/0.5-b48d156-r0/hello-isar_0.5.dsc 
exists in 
/workspace/build-auto/isar_master/1364/build/tmp/work/debian-bookworm-mipsel/hello-isar/0.5-b48d156-r0; 
copying to chroot
| I: NOTICE: Log filtering will replace 
'build/hello-isar-gmVcne/hello-isar-0.5' with '<<PKGBUILDDIR>>'
| I: NOTICE: Log filtering will replace 'build/hello-isar-gmVcne' with 
'<<BUILDDIR>>'
|
| 
+------------------------------------------------------------------------------+
| | Install package build 
dependencies                                           |
| 
+------------------------------------------------------------------------------+
|
|
| Setup apt archive
| -----------------
|
| E: 
/etc/schroot/chroot.d/isar-jenkins-6c34c167-bdc8-498f-828a-50cd88b412b0-2897105: 
Failed to stat file: No such file or directory
| E: Can't mkdir /<<RESOLVERDIR>>/sbuild-build-depends-main-dummy: No 
such file or directory
| W: Could not create build-depends dummy dir 
/<<RESOLVERDIR>>/sbuild-build-depends-main-dummyNo such file or 
directoryE: Setting up apt archive failed