Re: [PATCH v5] dpkg: Restore support for replacing pre-installed packages in sbuild-chroot

[Jan Kiszka <jan.kiszka@siemens.com>]

On 18.01.24 08:39, Uladzimir Bely wrote:
> On Thu, 2024-01-18 at 07:59 +0100, Jan Kiszka wrote:
>> On 18.01.24 07:34, Uladzimir Bely wrote:
>>> On Thu, 2024-01-18 at 09:28 +0300, Uladzimir Bely wrote:
>>>> On Tue, 2024-01-16 at 09:22 +0100, Jan Kiszka wrote:
>>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>>
>>>>> During the migration to sbuild, support for using self-built
>>>>> packages
>>>>> in
>>>>> the build environment was lost if those were already part of
>>>>> the
>>>>> sbuild-chroot. This restores it by adding --apt-distupgrade to
>>>>> the
>>>>> sbuild call. But that is not enough because sbuild will only
>>>>> upgrade
>>>>> packages from already configured sources, not those specified
>>>>> via
>>>>> --extra-repository. We therefore have to switch back to
>>>>> configuring
>>>>> isar-apt during sbuild-chroot creation.
>>>>>
>>>>> As rootfs_configure_isar_apt configures the isar repo under
>>>>> /isar-apt, we bind-mount the one in /home/builder/${PN} to that
>>>>> folder.
>>>>> Another difference is that we now need to run apt-get update
>>>>> explicitly,
>>>>> but only for isar-apt.
>>>>>
>>>>> And because the sbuild-chroots are now left behind with isar-
>>>>> apt
>>>>> configured, the configuration in imager_run can be removed
>>>>> (credits
>>>>> to
>>>>> Srinuvasan Arjunan).
>>>>>
>>>>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>>>>> ---
>>>>>
>>>>> Changes in v5:
>>>>>  - also use dpkg_schroot_create_configs for apt_fetch and
>>>>> apt_unpack
>>>>>    (while I wasn't able to reproduce the issue, I think this
>>>>> should
>>>>>    resolve the CI problems)
>>>>>
>>>>
>>>> Unfortunately, the picture in full CI didn't change. Even worse,
>>>> one
>>>> more job (NoCrossTest.test_nocross has failed with the similar
>>>> error.
>>>>
>>>> tmp/work/debian-bullseye-amd64/hello/1.0-
>>>> r0/temp/log.do_apt_unpack.1785250:
>>>> ```
>>>> DEBUG: Executing shell function do_apt_unpack
>>>> cp: cannot stat '/downloads/deb-src/debian-
>>>> bullseye/hello/hello_2.10-
>>>> 2+isar.debian.tar.xz': No such file or directory
>>>> cp: cannot stat '/downloads/deb-src/debian-
>>>> bullseye/hello/hello_2.10-
>>>> 2+isar.dsc': No such file or directory
>>>> ```
>>>>
>>>> I'm continuing debugging the issue.
>>>>
>>>
>>> Just one addition: previously (in v4) builds were failing in
>>> hello:apt_fetch, not they fail in hello:apt_unpack.
>>>
>>
>> Logs? The pattern above looks rather different from the previous
>> issue.
>>
>>> Also, now repro tests failed earlier - during base_apt:do_cache.
>>> E.g:
>>>
>>> tmp_repro_signed/work/debian-bullseye-arm64/base-apt/1.0-
>>> r0/temp/log.do_cache.2901916:
>>>
>>> ```
>>> ...
>>> Exporting indices...
>>> Error opening '/build/isar_ub_devel/359/downloads/deb-src/debian-
>>> bullseye/hello/hello_2.10-2+isar.dsc': No such file or directory
>>> There have been errors!
>>> WARNING: exit code 254 from a shell command.
>>> DEBUG: Python function do_cache finished
>>> ```
>>>
>>
>> How can this be reproduced locally in kas? That would be essential in
>> order to debug.
>>
>> Jan
>>
> 
> It's hard to reproduce the issue in some "short" form, but at least I
> found one quite fast way - steps int it are similar to what is do in
> unsigned repro test. In this test we do thre builds: normal build,
> cached build (after tmp cleanup), one more normal build (without
> cleanup).
> 
> The steps below do not fail with current 'next' branch of Isar, but
> produce hello:apt_fetch error with 'next + patch v5':
> 
> ```
> ./kas/kas-container menu
> # Just save and exit
> ./kas/kas-container shell
> # The rest done in kas shell
> # 1st build:
> bitbake hello
> # Build is OK
> # 2nd cached build:
> sudo rm -rf sstate-cache/ tmp/
> echo 'ISAR_USE_CACHED_BASE_REPO = "1"' >> conf/local.conf
> echo 'BB_NO_NETWORK = "1"' >> conf/local.conf
> bitbake hello
> # Build is OK
> # 3rd build with no cleanup
> echo 'ISAR_USE_CACHED_BASE_REPO = "0"' >> conf/local.conf
> echo 'BB_NO_NETWORK = "0"' >> conf/local.conf
> bitbake hello
> # Build fails in hello.bb:do_apt_fetch
> ```
> 
> I guess, other errors we observe in CI (e.g., in "do_apt_unpack") also
> have the similar nature, coming from sharing working directories
> between different tests that use the same targets but with different
> local.conf settings.

Ok, here comes the issue:

The hello recipe is trying to consume its own, previously built, source
package here. That is in fact inside the central isar-apt, but the local
copy does not contain it (yet). And the dependency chain we have does
not ensure that after the previous do_deploy_deb do_local_isarapt is
invalidated to re-run it before a succeeding recipe run.

One may argue why the package should consume its sources from isar-apt
rather than from upstream. To achieve that, we would need to create
preferences that filter these cases out for isar-apt. Not sure if that
will be generically possible. So we really need to ensure that the
missing task invalidation from above is somehow achieved. Or we need to
clean the caches more aggressively.

This issue is in fact "created" by fixing sbuild-schroot /wrt the use
cases of this patch. As these are more important than this corner case,
I would argue that the latter has to step back when we can't resolve
things perfectly.

Jan

-- 
Siemens AG, Technology
Linux Expert Center