Re: sbuild race with sporadic build breakages

[Jan Kiszka <jan.kiszka@siemens.com>]

On 24.06.24 10:00, Uladzimir Bely wrote:
> 
> 
> We were running CI with the patch from
> https://groups.google.com/g/isar-users/c/koNMYOyo-YM applied for
> several days and at some moment the issue happened again (with "heavy"
> NoCross test):
> 
> ```
> Creating /etc/schroot/chroot.d/isar-jenkins-7a841d41-9632-420f-a97d-
> bbf30e2732b9-
> 9f8d1f2d22de72cfbecc8e6ed8ceb6f59a829def67a2b9b7e2eb45a82af88ef2
> DEBUG: Shell function dpkg_schroot_create_configs finished
> DEBUG: Executing shell function dpkg_runbuild
> sbuild (Debian sbuild) 0.85.0 (04 January 2023) on turar.m.ilbers.de
> 
> +======================================================================
> ========+
> | libhello 0.3 (mipsel)                        Sat, 22 Jun 2024
> 12:34:22 +0000 |
> +======================================================================
> ========+
> 
> Package: libhello
> Version: 0.3
> Source Version: 0.3
> Distribution: isar
> Machine Architecture: amd64
> Host Architecture: mipsel
> Build Architecture: mipsel
> Build Type: binary
> 
> I: NOTICE: Log filtering will replace 'var/run/schroot/mount/isar-
> jenkins-7a841d41-9632-420f-a97d-bbf30e2732b9-
> 9f8d1f2d22de72cfbecc8e6ed8ceb6f59a829def67a2b9b7e2eb45a82af88ef2-
> 3ab2cd48-c709-4e9e-888f-a45fb2da713e' with '<<CHROOT>>'
> 
> +----------------------------------------------------------------------
> --------+
> | Chroot Setup Commands                                               
> |
> +----------------------------------------------------------------------
> --------+
> 
> 
> echo "Package: *\nPin: release n=isar\nPin-Priority: 1000" >
> /etc/apt/preferences.d/isar-apt
> -----------------------------------------------------------------------
> ---------------------
> 
> 
> I: Finished running 'echo "Package: *\nPin: release n=isar\nPin-
> Priority: 1000" > /etc/apt/preferences.d/isar-apt'.
> 
> echo "APT::Get::allow-downgrades 1;" > /etc/apt/apt.conf.d/50isar-apt
> ---------------------------------------------------------------------
> 
> 
> I: Finished running 'echo "APT::Get::allow-downgrades 1;" >
> /etc/apt/apt.conf.d/50isar-apt'.
> 
> rm -f /var/log/dpkg.log
> -----------------------
> 
> 
> I: Finished running 'rm -f /var/log/dpkg.log'.
> 
> mkdir -p /var/cache/apt/archives
> --------------------------------
> 
> 
> I: Finished running 'mkdir -p /var/cache/apt/archives'.
> 
> find /home/builder/libhello/rootfs/var/cache/apt/archives -maxdepth 1 -
> name '*.deb' -exec ln -t /var/cache/apt/archives/ -sf {} +
> -----------------------------------------------------------------------
> ----------------------------------------------------------
> 
> 
> I: Finished running 'find
> /home/builder/libhello/rootfs/var/cache/apt/archives -maxdepth 1 -name
> '*.deb' -exec ln -t /var/cache/apt/archives/ -sf {} +'.
> 
> apt-get update -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" -
> o Dir::Etc::SourceParts="-" -o APT::Get::List-Cleanup="0"
> -----------------------------------------------------------------------
> ---------------------------------------------------------
> 
> Get:1 file:/isar-apt isar InRelease
> Ign:1 file:/isar-apt isar InRelease
> Get:2 file:/isar-apt isar Release [5006 B]
> Get:2 file:/isar-apt isar Release [5006 B]
> Get:3 file:/isar-apt isar Release.gpg
> Ign:3 file:/isar-apt isar Release.gpg
> Reading package lists...
> 
> I: Finished running 'apt-get update -o
> Dir::Etc::SourceList="sources.list.d/isar-apt.list" -o
> Dir::Etc::SourceParts="-" -o APT::Get::List-Cleanup="0"'.
> 
> Finished processing commands.
> -----------------------------------------------------------------------
> ---------
> I: NOTICE: Log filtering will replace 'build/libhello-BBthl5/resolver-
> X1hxT7' with '<<RESOLVERDIR>>'
> 
> +----------------------------------------------------------------------
> --------+
> | Update chroot                                                       
> |
> +----------------------------------------------------------------------
> --------+
> 
> Reading package lists...
> Building dependency tree...
> Reading state information...
> Calculating upgrade...
> 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
> 
> +----------------------------------------------------------------------
> --------+
> | Fetch source files                                                  
> |
> +----------------------------------------------------------------------
> --------+
> 
> 
> Local sources
> -------------
> 
> /workspace/build/isar_ub_devel/436/build/tmp/work/debian-bookworm-
> mipsel/libhello/0.3-f4a5669-r0/libhello_0.3.dsc exists in
> /workspace/build/isar_ub_devel/436/build/tmp/work/debian-bookworm-
> mipsel/libhello/0.3-f4a5669-r0; copying to chroot
> I: NOTICE: Log filtering will replace 'build/libhello-BBthl5/libhello-
> 0.3' with '<<PKGBUILDDIR>>'
> I: NOTICE: Log filtering will replace 'build/libhello-BBthl5' with
> '<<BUILDDIR>>'
> 
> +----------------------------------------------------------------------
> --------+
> | Install package build dependencies                                  
> |
> +----------------------------------------------------------------------
> --------+
> 
> 
> Setup apt archive
> -----------------
> 
> E: /var/lib/schroot/session/isar-jenkins-7a841d41-9632-420f-a97d-
> bbf30e2732b9-
> 8641cd44af3718c57bd0c9e341b9dd285c91d116b7bb1d5b93a87a4e4c043d01-
> 0f576f86-4a3b-4d43-a366-f14782e97097: Failed to open file: No such file
> or directory
> E: Can't chmod /<<RESOLVERDIR>> to 0770: No such file or directory
> E: Failed to set 0770 permissions on dummy package dir
> E: Setting up apt archive failed
> ```
> 
> And that's what we see in logs:
> 
> ```
> build/tmp$ grep -R f14782e97097
> work/debian-bookworm-mipsel/libhello/0.3-f4a5669-
> r0/temp/log.do_dpkg_build:E: /var/lib/schroot/session/isar-jenkins-
> 7a841d41-9632-420f-a97d-bbf30e2732b9-
> 8641cd44af3718c57bd0c9e341b9dd285c91d116b7bb1d5b93a87a4e4c043d01-
> 0f576f86-4a3b-4d43-a366-f14782e97097: Failed to open file: No such file
> or directory
> work/debian-bookworm-mipsel/libhello/0.3-f4a5669-
> r0/temp/log.do_dpkg_build.694689:E: /var/lib/schroot/session/isar-
> jenkins-7a841d41-9632-420f-a97d-bbf30e2732b9-
> 8641cd44af3718c57bd0c9e341b9dd285c91d116b7bb1d5b93a87a4e4c043d01-
> 0f576f86-4a3b-4d43-a366-f14782e97097: Failed to open file: No such file
> or directory
> work/debian-bookworm-amd64/kselftest/5.6.13-
> r0/temp/log.do_dpkg_build.369632:I: NOTICE: Log filtering will replace
> 'var/run/schroot/mount/isar-jenkins-7a841d41-9632-420f-a97d-
> bbf30e2732b9-
> 8641cd44af3718c57bd0c9e341b9dd285c91d116b7bb1d5b93a87a4e4c043d01-
> 0f576f86-4a3b-4d43-a366-f14782e97097' with '<<CHROOT>>'
> work/debian-bookworm-amd64/kselftest/5.6.13-
> r0/temp/log.do_dpkg_build.369632:SCHROOT_SESSION_ID=isar-jenkins-
> 7a841d41-9632-420f-a97d-bbf30e2732b9-
> 8641cd44af3718c57bd0c9e341b9dd285c91d116b7bb1d5b93a87a4e4c043d01-
> 0f576f86-4a3b-4d43-a366-f14782e97097
> ```
> 
> For some reason, when building "libhello" package for "mipsel"
> architecture, sbuild tried to use existing directory, that "kselftest"
> package form "amd64" used.
> 
> So, replacing pid with task hashsum does not help with this sporadic
> issue (but reduces readability of log files).
> 
> We'll continue debugging the root cause of the issue.
> 

Interesting - at least we ruled out one potential cause this way.

Thanks,
Jan

-- 
Siemens AG, Technology
Linux Expert Center