Re: Cross-build issues over bookworm host for non-bookworm targets

[Jan Kiszka <jan.kiszka@siemens.com>]

On 06.06.23 07:06, Uladzimir Bely wrote:
> On Mon, 2023-06-05 at 06:40 +0200, Jan Kiszka wrote:
>> On 02.06.23 18:59, Uladzimir Bely wrote:
>>> On Fri, 2023-06-02 at 10:30 +0200, 'Jan Kiszka' via isar-users
>>> wrote:
>>>> Hi all,
>>>>
>>>> the kas-isar:next build container is now based on bookworm, and
>>>> that 
>>>> seems to reveal some issues around cross-building over that host.
>>>> I've 
>>>> not reproduced that with two layers, meta-iot2050 and isar-cip-
>>>> core.
>>>> The 
>>>> latter builds fine natively, e.g. qemu-amd64, but it fails once 
>>>> switching to qemu-arm64 e.g. BUT: bookworm as target is fine
>>>> again.
>>>>
>>>> Some exemplary splat:
>>>>
>>>> ERROR: Logfile of failure stored in: /build/tmp/work/cip-core-
>>>> bullseye-arm64/sshd-regen-keys/0.4-
>>>> r0/temp/log.do_dpkg_build.14055
>>>> Log data follows:
>>>>> DEBUG: Executing python function sstate_task_prefunc
>>>>> DEBUG: Python function sstate_task_prefunc finished
>>>>> DEBUG: Executing python function do_dpkg_build
>>>>> DEBUG: Executing shell function schroot_create_configs
>>>>> DEBUG: Shell function schroot_create_configs finished
>>>>> DEBUG: Executing shell function dpkg_runbuild
>>>>> sbuild (Debian sbuild) 0.85.0 (04 January 2023) on 76d4480b6118
>>>>>
>>>>> +==============================================================
>>>>> ====
>>>>> ============+
>>>>>> sshd-regen-keys 0.4 (arm64)                  Fri, 02 Jun 2023
>>>>>> 08:27:29 +0000 |
>>>>> +==============================================================
>>>>> ====
>>>>> ============+
>>>>>
>>>>> Package: sshd-regen-keys
>>>>> Version: 0.4
>>>>> Source Version: 0.4
>>>>> Distribution: isar
>>>>> Machine Architecture: amd64
>>>>> Host Architecture: arm64
>>>>> Build Architecture: amd64
>>>>> Build Profiles: cross
>>>>> Build Type: binary
>>>>>
>>>>> I: NOTICE: Log filtering will replace
>>>>> 'var/run/schroot/mount/isar-
>>>>> builder-80876582-28a6-4cef-89cf-0c1aacac137d-14055-caaa44a8-
>>>>> 48f9-
>>>>> 4b91-bf10-b777cbca86a3' with '<<CHROOT>>'
>>>>>
>>>>> +--------------------------------------------------------------
>>>>> ----
>>>>> ------------+
>>>>>> Chroot Setup
>>>>>> Commands                                                     
>>>>>>    |
>>>>> +--------------------------------------------------------------
>>>>> ----
>>>>> ------------+
>>>>>
>>>>>
>>>>> echo "Package: *\nPin: release n=isar\nPin-Priority: 1000" >
>>>>> /etc/apt/preferences.d/isar-apt
>>>>> ---------------------------------------------------------------
>>>>> ----
>>>>> -------------------------
>>>>>
>>>>>
>>>>> I: Finished running 'echo "Package: *\nPin: release
>>>>> n=isar\nPin-
>>>>> Priority: 1000" > /etc/apt/preferences.d/isar-apt'.
>>>>>
>>>>> echo "APT::Get::allow-downgrades 1;" >
>>>>> /etc/apt/apt.conf.d/50isar-
>>>>> apt
>>>>> ---------------------------------------------------------------
>>>>> ----
>>>>> --
>>>>>
>>>>>
>>>>> I: Finished running 'echo "APT::Get::allow-downgrades 1;" >
>>>>> /etc/apt/apt.conf.d/50isar-apt'.
>>>>>
>>>>> rm -f /var/log/dpkg.log
>>>>> -----------------------
>>>>>
>>>>>
>>>>> I: Finished running 'rm -f /var/log/dpkg.log'.
>>>>>
>>>>> ln -sf /home/builder/sshd-regen-
>>>>> keys/rootfs/var/cache/apt/archives/*.deb -t
>>>>> /var/cache/apt/archives/ || :
>>>>> ---------------------------------------------------------------
>>>>> ----
>>>>> --------------------------------------
>>>>>
>>>>>
>>>>> I: Finished running 'ln -sf /home/builder/sshd-regen-
>>>>> keys/rootfs/var/cache/apt/archives/*.deb -t
>>>>> /var/cache/apt/archives/ || :'.
>>>>>
>>>>> Finished processing commands.
>>>>> ---------------------------------------------------------------
>>>>> ----
>>>>> -------------
>>>>> I: NOTICE: Log filtering will replace 'build/sshd-regen-keys-
>>>>> AuXR83/resolver-Ms1W52' with '<<RESOLVERDIR>>'
>>>>>
>>>>> +--------------------------------------------------------------
>>>>> ----
>>>>> ------------+
>>>>>> Update
>>>>>> chroot                                                       
>>>>>>     
>>>>>>      |
>>>>> +--------------------------------------------------------------
>>>>> ----
>>>>> ------------+
>>>>>
>>>>> Reading package lists...
>>>>> Building dependency tree...
>>>>> Reading state information...
>>>>> Calculating upgrade...
>>>>> 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
>>>>>
>>>>> +--------------------------------------------------------------
>>>>> ----
>>>>> ------------+
>>>>>> Fetch source
>>>>>> files                                                        
>>>>>>    |
>>>>> +--------------------------------------------------------------
>>>>> ----
>>>>> ------------+
>>>>>
>>>>>
>>>>> Local sources
>>>>> -------------
>>>>>
>>>>> /build/tmp/work/cip-core-bullseye-arm64/sshd-regen-keys/0.4-
>>>>> r0/sshd-regen-keys_0.4.dsc exists in /build/tmp/work/cip-core-
>>>>> bullseye-arm64/sshd-regen-keys/0.4-r0; copying to chroot
>>>>> I: NOTICE: Log filtering will replace 'build/sshd-regen-keys-
>>>>> AuXR83/sshd-regen-keys-0.4' with '<<PKGBUILDDIR>>'
>>>>> I: NOTICE: Log filtering will replace 'build/sshd-regen-keys-
>>>>> AuXR83' with '<<BUILDDIR>>'
>>>>>
>>>>> +--------------------------------------------------------------
>>>>> ----
>>>>> ------------+
>>>>>> Install package build
>>>>>> dependencies                                           |
>>>>> +--------------------------------------------------------------
>>>>> ----
>>>>> ------------+
>>>>>
>>>>>
>>>>> Setup apt archive
>>>>> -----------------
>>>>>
>>>>> Merged Build-Depends: debhelper (>= 10), build-essential:amd64,
>>>>> fakeroot:amd64, crossbuild-essential-arm64:amd64, libc-
>>>>> dev:arm64,
>>>>> libstdc++-dev:arm64
>>>>> Filtered Build-Depends: debhelper (>= 10), build-
>>>>> essential:amd64,
>>>>> fakeroot:amd64, crossbuild-essential-arm64:amd64, libc-
>>>>> dev:arm64,
>>>>> libstdc++-dev:arm64
>>>>> dpkg-deb: building package 'sbuild-build-depends-main-dummy' in
>>>>> '/<<RESOLVERDIR>>/apt_archive/sbuild-build-depends-main-
>>>>> dummy.deb'.
>>>>> Ign:1 copy:/<<RESOLVERDIR>>/apt_archive ./ InRelease
>>>>> Get:2 copy:/<<RESOLVERDIR>>/apt_archive ./ Release [609 B]
>>>>> Ign:3 copy:/<<RESOLVERDIR>>/apt_archive ./ Release.gpg
>>>>> Get:4 copy:/<<RESOLVERDIR>>/apt_archive ./ Sources [683 B]
>>>>> Get:5 copy:/<<RESOLVERDIR>>/apt_archive ./ Packages [726 B]
>>>>> Fetched 2018 B in 0s (0 B/s)
>>>>> Reading package lists...
>>>>> Get:1 file:/home/builder/sshd-regen-keys/isar-apt/cip-core-
>>>>> bullseye-arm64/apt/cip-core-bullseye isar InRelease
>>>>> Ign:1 file:/home/builder/sshd-regen-keys/isar-apt/cip-core-
>>>>> bullseye-arm64/apt/cip-core-bullseye isar InRelease
>>>>> Get:2 file:/home/builder/sshd-regen-keys/isar-apt/cip-core-
>>>>> bullseye-arm64/apt/cip-core-bullseye isar Release [4994 B]
>>>>> Get:2 file:/home/builder/sshd-regen-keys/isar-apt/cip-core-
>>>>> bullseye-arm64/apt/cip-core-bullseye isar Release [4994 B]
>>>>> Get:3 file:/home/builder/sshd-regen-keys/isar-apt/cip-core-
>>>>> bullseye-arm64/apt/cip-core-bullseye isar Release.gpg
>>>>> Ign:3 file:/home/builder/sshd-regen-keys/isar-apt/cip-core-
>>>>> bullseye-arm64/apt/cip-core-bullseye isar Release.gpg
>>>>> Reading package lists...
>>>>> Reading package lists...
>>>>>
>>>>> Install main build dependencies (apt-based resolver)
>>>>> ----------------------------------------------------
>>>>>
>>>>> Installing build dependencies
>>>>> Reading package lists...
>>>>> Building dependency tree...
>>>>> Reading state information...
>>>>> /usr/lib/apt/solvers/apt: /lib/x86_64-linux-gnu/libc.so.6:
>>>>> version
>>>>> `GLIBC_2.34' not found (required by /usr/lib/apt/solvers/apt)
>>>>> Execute external solver...
>>>>> E: Write error - write (32: Broken pipe)
>>>>> E: Write error - write (32: Broken pipe)
>>>>> E: Sub-process sbuild-cross-resolver received signal 13.
>>>>> apt-get failed.
>>>>> E: Package installation failed
>>>>> Not removing build depends: cloned chroot in use
>>>>
>>>>
>>>> "/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not
>>>> found",
>>>> that looks suspicious.
>>>>
>>>> Are we leaking something from the host into the sbuild env that
>>>> falls
>>>> apart here? Are we missing a dependency for the sbuild env? Or is
>>>> there 
>>>> a fundamental incompatibility in bookworm? 
>>>>
>>>> Jan
>>>>
>>>> -- 
>>>> Siemens AG, Technology
>>>> Competence Center Embedded Linux
>>>>
>>>
>>> Adding "apt-utils" to SBUILD_CHROOT_PREINSTALL_COMMON  in
>>> `meta/recipes-devtools/sbuild-chroot` seems to solve the issue.
>>>
>>> This package provides `/usr/lib/apt/solvers/apt` in sbuild-chroot. 
>>
>> Can you also explain why adding that resolves the specific error
>> above?
>> So far, apt-utils was not needed. Is some script different that comes
>> with the host sbuild package? I'm still concerned that some binary of
>> the host leaks and would like to see that ruled out.
>>
>> Jan
>>
> 
> I've looked through the issue and that's what I found.
> 
> It seems to be related to new apt solver they use for cross-building
> (addded in https://salsa.debian.org/debian/sbuild/-/commit/696e963).
> 
> For older sbuild (e.g., bullseye-based sbuild-chroot) we have the only
> `dump` solver belonging to apt package:
> 
> ```
> builder@c8c24e2a0203:/build$ ls -1 /usr/lib/apt/solvers/
> dump
> builder@c8c24e2a0203:/build$ dpkg -S /usr/lib/apt/solvers/*
> apt: /usr/lib/apt/solvers/dump
> ```
> 
> With newer sbuild (e.g., for ookworm-based sbuild-chroot) we have three
> solvers availabler:
> 
> ```
> builder@834442924aa6:/build$ dpkg -S /usr/lib/apt/solvers/*
> apt-utils: /usr/lib/apt/solvers/apt
> apt: /usr/lib/apt/solvers/dump
> sbuild: /usr/lib/apt/solvers/sbuild-cross-resolver
> ```
> 
> The last one is enabled when cross-building. And since new sbuild has a
> strict dependency on apt-utils (ia "sbuild => libsbuild-perl => apt-
> utils" chain; in bullseye it was just "recommended"), it makes us add
> apt-utils package inside cross-build environment.

OK, sounds consistent to me, thanks. Will you send a patch?

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux