Re: Cross-build issues over bookworm host for non-bookworm targets

[Uladzimir Bely <ubely@ilbers.de>]

On Mon, 2023-06-05 at 06:40 +0200, Jan Kiszka wrote:
> On 02.06.23 18:59, Uladzimir Bely wrote:
> > On Fri, 2023-06-02 at 10:30 +0200, 'Jan Kiszka' via isar-users
> > wrote:
> > > Hi all,
> > > 
> > > the kas-isar:next build container is now based on bookworm, and
> > > that 
> > > seems to reveal some issues around cross-building over that host.
> > > I've 
> > > not reproduced that with two layers, meta-iot2050 and isar-cip-
> > > core.
> > > The 
> > > latter builds fine natively, e.g. qemu-amd64, but it fails once 
> > > switching to qemu-arm64 e.g. BUT: bookworm as target is fine
> > > again.
> > > 
> > > Some exemplary splat:
> > > 
> > > ERROR: Logfile of failure stored in: /build/tmp/work/cip-core-
> > > bullseye-arm64/sshd-regen-keys/0.4-
> > > r0/temp/log.do_dpkg_build.14055
> > > Log data follows:
> > > > DEBUG: Executing python function sstate_task_prefunc
> > > > DEBUG: Python function sstate_task_prefunc finished
> > > > DEBUG: Executing python function do_dpkg_build
> > > > DEBUG: Executing shell function schroot_create_configs
> > > > DEBUG: Shell function schroot_create_configs finished
> > > > DEBUG: Executing shell function dpkg_runbuild
> > > > sbuild (Debian sbuild) 0.85.0 (04 January 2023) on 76d4480b6118
> > > > 
> > > > +==============================================================
> > > > ====
> > > > ============+
> > > > > sshd-regen-keys 0.4 (arm64)                  Fri, 02 Jun 2023
> > > > > 08:27:29 +0000 |
> > > > +==============================================================
> > > > ====
> > > > ============+
> > > > 
> > > > Package: sshd-regen-keys
> > > > Version: 0.4
> > > > Source Version: 0.4
> > > > Distribution: isar
> > > > Machine Architecture: amd64
> > > > Host Architecture: arm64
> > > > Build Architecture: amd64
> > > > Build Profiles: cross
> > > > Build Type: binary
> > > > 
> > > > I: NOTICE: Log filtering will replace
> > > > 'var/run/schroot/mount/isar-
> > > > builder-80876582-28a6-4cef-89cf-0c1aacac137d-14055-caaa44a8-
> > > > 48f9-
> > > > 4b91-bf10-b777cbca86a3' with '<<CHROOT>>'
> > > > 
> > > > +--------------------------------------------------------------
> > > > ----
> > > > ------------+
> > > > > Chroot Setup
> > > > > Commands                                                     
> > > > >    |
> > > > +--------------------------------------------------------------
> > > > ----
> > > > ------------+
> > > > 
> > > > 
> > > > echo "Package: *\nPin: release n=isar\nPin-Priority: 1000" >
> > > > /etc/apt/preferences.d/isar-apt
> > > > ---------------------------------------------------------------
> > > > ----
> > > > -------------------------
> > > > 
> > > > 
> > > > I: Finished running 'echo "Package: *\nPin: release
> > > > n=isar\nPin-
> > > > Priority: 1000" > /etc/apt/preferences.d/isar-apt'.
> > > > 
> > > > echo "APT::Get::allow-downgrades 1;" >
> > > > /etc/apt/apt.conf.d/50isar-
> > > > apt
> > > > ---------------------------------------------------------------
> > > > ----
> > > > --
> > > > 
> > > > 
> > > > I: Finished running 'echo "APT::Get::allow-downgrades 1;" >
> > > > /etc/apt/apt.conf.d/50isar-apt'.
> > > > 
> > > > rm -f /var/log/dpkg.log
> > > > -----------------------
> > > > 
> > > > 
> > > > I: Finished running 'rm -f /var/log/dpkg.log'.
> > > > 
> > > > ln -sf /home/builder/sshd-regen-
> > > > keys/rootfs/var/cache/apt/archives/*.deb -t
> > > > /var/cache/apt/archives/ || :
> > > > ---------------------------------------------------------------
> > > > ----
> > > > --------------------------------------
> > > > 
> > > > 
> > > > I: Finished running 'ln -sf /home/builder/sshd-regen-
> > > > keys/rootfs/var/cache/apt/archives/*.deb -t
> > > > /var/cache/apt/archives/ || :'.
> > > > 
> > > > Finished processing commands.
> > > > ---------------------------------------------------------------
> > > > ----
> > > > -------------
> > > > I: NOTICE: Log filtering will replace 'build/sshd-regen-keys-
> > > > AuXR83/resolver-Ms1W52' with '<<RESOLVERDIR>>'
> > > > 
> > > > +--------------------------------------------------------------
> > > > ----
> > > > ------------+
> > > > > Update
> > > > > chroot                                                       
> > > > >     
> > > > >      |
> > > > +--------------------------------------------------------------
> > > > ----
> > > > ------------+
> > > > 
> > > > Reading package lists...
> > > > Building dependency tree...
> > > > Reading state information...
> > > > Calculating upgrade...
> > > > 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
> > > > 
> > > > +--------------------------------------------------------------
> > > > ----
> > > > ------------+
> > > > > Fetch source
> > > > > files                                                        
> > > > >    |
> > > > +--------------------------------------------------------------
> > > > ----
> > > > ------------+
> > > > 
> > > > 
> > > > Local sources
> > > > -------------
> > > > 
> > > > /build/tmp/work/cip-core-bullseye-arm64/sshd-regen-keys/0.4-
> > > > r0/sshd-regen-keys_0.4.dsc exists in /build/tmp/work/cip-core-
> > > > bullseye-arm64/sshd-regen-keys/0.4-r0; copying to chroot
> > > > I: NOTICE: Log filtering will replace 'build/sshd-regen-keys-
> > > > AuXR83/sshd-regen-keys-0.4' with '<<PKGBUILDDIR>>'
> > > > I: NOTICE: Log filtering will replace 'build/sshd-regen-keys-
> > > > AuXR83' with '<<BUILDDIR>>'
> > > > 
> > > > +--------------------------------------------------------------
> > > > ----
> > > > ------------+
> > > > > Install package build
> > > > > dependencies                                           |
> > > > +--------------------------------------------------------------
> > > > ----
> > > > ------------+
> > > > 
> > > > 
> > > > Setup apt archive
> > > > -----------------
> > > > 
> > > > Merged Build-Depends: debhelper (>= 10), build-essential:amd64,
> > > > fakeroot:amd64, crossbuild-essential-arm64:amd64, libc-
> > > > dev:arm64,
> > > > libstdc++-dev:arm64
> > > > Filtered Build-Depends: debhelper (>= 10), build-
> > > > essential:amd64,
> > > > fakeroot:amd64, crossbuild-essential-arm64:amd64, libc-
> > > > dev:arm64,
> > > > libstdc++-dev:arm64
> > > > dpkg-deb: building package 'sbuild-build-depends-main-dummy' in
> > > > '/<<RESOLVERDIR>>/apt_archive/sbuild-build-depends-main-
> > > > dummy.deb'.
> > > > Ign:1 copy:/<<RESOLVERDIR>>/apt_archive ./ InRelease
> > > > Get:2 copy:/<<RESOLVERDIR>>/apt_archive ./ Release [609 B]
> > > > Ign:3 copy:/<<RESOLVERDIR>>/apt_archive ./ Release.gpg
> > > > Get:4 copy:/<<RESOLVERDIR>>/apt_archive ./ Sources [683 B]
> > > > Get:5 copy:/<<RESOLVERDIR>>/apt_archive ./ Packages [726 B]
> > > > Fetched 2018 B in 0s (0 B/s)
> > > > Reading package lists...
> > > > Get:1 file:/home/builder/sshd-regen-keys/isar-apt/cip-core-
> > > > bullseye-arm64/apt/cip-core-bullseye isar InRelease
> > > > Ign:1 file:/home/builder/sshd-regen-keys/isar-apt/cip-core-
> > > > bullseye-arm64/apt/cip-core-bullseye isar InRelease
> > > > Get:2 file:/home/builder/sshd-regen-keys/isar-apt/cip-core-
> > > > bullseye-arm64/apt/cip-core-bullseye isar Release [4994 B]
> > > > Get:2 file:/home/builder/sshd-regen-keys/isar-apt/cip-core-
> > > > bullseye-arm64/apt/cip-core-bullseye isar Release [4994 B]
> > > > Get:3 file:/home/builder/sshd-regen-keys/isar-apt/cip-core-
> > > > bullseye-arm64/apt/cip-core-bullseye isar Release.gpg
> > > > Ign:3 file:/home/builder/sshd-regen-keys/isar-apt/cip-core-
> > > > bullseye-arm64/apt/cip-core-bullseye isar Release.gpg
> > > > Reading package lists...
> > > > Reading package lists...
> > > > 
> > > > Install main build dependencies (apt-based resolver)
> > > > ----------------------------------------------------
> > > > 
> > > > Installing build dependencies
> > > > Reading package lists...
> > > > Building dependency tree...
> > > > Reading state information...
> > > > /usr/lib/apt/solvers/apt: /lib/x86_64-linux-gnu/libc.so.6:
> > > > version
> > > > `GLIBC_2.34' not found (required by /usr/lib/apt/solvers/apt)
> > > > Execute external solver...
> > > > E: Write error - write (32: Broken pipe)
> > > > E: Write error - write (32: Broken pipe)
> > > > E: Sub-process sbuild-cross-resolver received signal 13.
> > > > apt-get failed.
> > > > E: Package installation failed
> > > > Not removing build depends: cloned chroot in use
> > > 
> > > 
> > > "/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not
> > > found",
> > > that looks suspicious.
> > > 
> > > Are we leaking something from the host into the sbuild env that
> > > falls
> > > apart here? Are we missing a dependency for the sbuild env? Or is
> > > there 
> > > a fundamental incompatibility in bookworm? 
> > > 
> > > Jan
> > > 
> > > -- 
> > > Siemens AG, Technology
> > > Competence Center Embedded Linux
> > > 
> > 
> > Adding "apt-utils" to SBUILD_CHROOT_PREINSTALL_COMMON  in
> > `meta/recipes-devtools/sbuild-chroot` seems to solve the issue.
> > 
> > This package provides `/usr/lib/apt/solvers/apt` in sbuild-chroot. 
> 
> Can you also explain why adding that resolves the specific error
> above?
> So far, apt-utils was not needed. Is some script different that comes
> with the host sbuild package? I'm still concerned that some binary of
> the host leaks and would like to see that ruled out.
> 
> Jan
> 

I've looked through the issue and that's what I found.

It seems to be related to new apt solver they use for cross-building
(addded in https://salsa.debian.org/debian/sbuild/-/commit/696e963).

For older sbuild (e.g., bullseye-based sbuild-chroot) we have the only
`dump` solver belonging to apt package:

```
builder@c8c24e2a0203:/build$ ls -1 /usr/lib/apt/solvers/
dump
builder@c8c24e2a0203:/build$ dpkg -S /usr/lib/apt/solvers/*
apt: /usr/lib/apt/solvers/dump
```

With newer sbuild (e.g., for ookworm-based sbuild-chroot) we have three
solvers availabler:

```
builder@834442924aa6:/build$ dpkg -S /usr/lib/apt/solvers/*
apt-utils: /usr/lib/apt/solvers/apt
apt: /usr/lib/apt/solvers/dump
sbuild: /usr/lib/apt/solvers/sbuild-cross-resolver
```

The last one is enabled when cross-building. And since new sbuild has a
strict dependency on apt-utils (ia "sbuild => libsbuild-perl => apt-
utils" chain; in bullseye it was just "recommended"), it makes us add
apt-utils package inside cross-build environment.

Uladzimir