Re: base-apt: Inconsistent priorities between apt-cache and .deb data break offline builds

[Anton Mikanovich <amikan@ilbers.de>]

01.07.2026 11:56, MOESSBAUER, Felix wrote:
> On Wed, 2026-07-01 at 10:36 +0300, Anton Mikanovich wrote:
>> 30.06.2026 20:26, MOESSBAUER, Felix wrote:
>>> On Tue, 2026-06-30 at 18:36 +0300, Anton Mikanovich wrote:
>>>> On 30/06/2026 16:24, 'MOESSBAUER, Felix' via isar-users wrote:
>>>>> Dear isar-devs,
>>>>>
>>>>> we noticed that cross-architecture offline builds are currently broken
>>>>> and single-architecture offline builds differ from the online
>>>>> variant. One failure pattern is the following, whereby the offline
>>>>> isar-mmdebstrap-host fails with an error message like this:
>>>>>
>>>>> The following packages have unmet dependencies:
>>>>>       libtext-wrapi18n-perl : Depends: libtext-charwidth-perl but it is not
>>>>> installable
>>>>> E: Unable to correct problems, you have held broken packages.
>>>>> E: The following information from --solver 3.0 may provide additional
>>>>> context:
>>>>>         Unable to satisfy dependencies. Reached two conflicting decisions:
>>>>>         1. libtext-wrapi18n-perl:amd64=0.06-10 is selected for install
>>>>>         2. libtext-wrapi18n-perl:amd64 Depends libtext-charwidth-perl
>>>>>            but none of the choices are installable:
>>>>>            [no choices]
>>>>>
>>>>> The root cause for this is that we ignore the upstream's repo priority
>>>>> overrides when creating the base-apt: reprepro includedeb records the
>>>>> priority from the .deb's own control field, and that becomes the
>>>>> priority published in base-apt's Packages index.
>>>>>
>>>>> When these priorities differ, the set of packages installed in offline
>>>>> mode are different from the ones in online mode. This per-se is plain
>>>>> wrong, but only surfaces if a now-required arch=all package depends on
>>>>> its architecture specific variant which never has been fetched (because
>>>>> the online bootstrap had access to the prio overrides). For non-cross
>>>>> builds the result is also different, which can be checked by comparing
>>>>> the SBOMs.
>>>>>
>>>>> There are HACKY ways to re-create the upstream priorities, but IMHO our
>>>>> approach for offline builds is problematic and should be changed:
>>>>> Instead of explicitly recreating the base-apt repo, we better just
>>>>> preserve the upstream apt lists and inject our downloads/deb dir into
>>>>> the apt cache that is used by the build. This would allow us to
>>>>> completely drop the base-apt.
>>>> Issues like that supposed to be fixed by "Improving base-apt usage"
>>>> patchset.
>>> This rings a bell. Looks like the series could need a revival.
>>>
>>>> It should address storing only needed packages, while DL_DIR is just a
>>>> storage
>>>> for everything without storing any real packages state in it.
>>>> The idea is to have base-apt as an input, while isar-apt is used as an
>>>> output.
>>> I just checked the series, but this issue is not solved by it, as the
>>> apt lists are not cached but only the packages. This is exactly what
>>> breaks here: The packages have a different priority field than the apt
>>> list.
>>>
>>> I'm wondering if a simpler solution is a pull-through cache, like many
>>> people use for cached building of containers (like here [1]). That
>>> let's apt take care of the caching instead of the surrounding build
>>> environment.
>>>
>>> [1]
>>> https://github.com/siemens/kas/blob/81bab5294be29de4b64adedce962d36184bfc66f/Dockerfile#L37
>>>
>>>> We will look into the issue, starting from reproducing it.
>>> It's easy to reproduce with these steps. You can even further reduce
>>> the set of packages to the bare minimum to reproduce faster.
>> Turns out not so easy: tried twice on clean setup with no luck so far.
>> Are there any other options or changes?
> Hi, you must start with an empty downloads folder. The libtext-
> charwidth-perl_0.04-11+b4_amd64.deb indicates that the isar-image-base
> was built for amd64.
No, it was fetched during cross-building of any of hello/hello-isar/cowsay
packages, which actually mask this issue. So disabling default packages is
needed not just to speedup the build, but to unmask the issue.
Now I can reproduce it.
>
> For debugging, I recommend to just build the bootstrap target for amd64
> and check that the libtext-charwidth-perl_0.04-11+b4_arm64.deb is NOT
> in the downloads dir. Once that is confirmed, please retry the steps
> from below.
>
> Felix
>
>> Here are debs in downloads after the build (on next b90b722f, default
>> debian-trixie):
>> libtext-wrapi18n-perl_0.06-10_all.deb
>> libtext-charwidth-perl_0.04-11+b4_amd64.deb
>> libtext-charwidth-perl_0.04-11+b4_arm64.deb
>>>>> Reproducer:
>>>>>
>>>>> 1. kas-container menu, select qemuarm64 machine, build
>>>>> 2. mv build build-online && mkdir build && cp -rf build-
>>>>> online/downloads build/
>>>>> 3. kas-container build .config.yaml:offline.yml
>>>>>
>>>>> offline.yml:
>>>>> header:
>>>>>        version: 14
>>>>> local_conf_header:
>>>>>        offline: |
>>>>>          ISAR_USE_CACHED_BASE_REPO = "1"
>>>>>          BB_NO_NETWORK = "1"
>>>> Do we need such an option in kas menu?
>>> Probably? :) At least I did not see this option, but the variables are
>>> documented.
>>>
>>> Felix
>> --
>> You received this message because you are subscribed to the Google Groups "isar-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
>> To view this discussion visit https://groups.google.com/d/msgid/isar-users/6de97274-3f2c-4b62-a7e9-82f6e1f2c338%40ilbers.de.

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/6319798c-ee48-4d18-9f3b-c32e0569ab9c%40ilbers.de.