Re: [PATCH 00/13] More build time reductions

["'MOESSBAUER, Felix' via isar-users" <isar-users@googlegroups.com>]

On Fri, 2026-10-16 at 14:23 +0200, Zhihang Wei wrote:
> On 10/2/26 16:49, 'Felix Moessbauer' via isar-users wrote:
> > This is merely a dump of the remaining patches mentioned
> > in my talk "Isar build time reduction" at the isar community
> > event in Prague. The series itself is fast-ci tested and also passes
> > most (all?, unrelated failures) of the tests on full CI.
> > 
> > The biggest impact has the switch to the host-apt (which now also
> > is compatible with base-apt), which allows us to not have apt
> > in the image at all.
> > 
> > The series goes on top of "Model isar-exclude-docs as rootfs feature".
> > 
> > Best regards,
> > Felix Moessbauer
> > 
> > Felix Moessbauer (13):
> >    debianize: services are named after BPN not PN
> >    dpkg-raw: do not leak package variants into debian package
> >    debian templates: name package after BPN instead of PN
> >    linux-distro: provide fake packages for native variant of packages
> >    dpkg-source: let the native package variant own the source package
> >    do not assert on legitimate re-execution of tasks
> >    bitbake: backport cooker: use BB_HASHSERVE_DB_DIR for hash server
> >      database location
> >    set default sstate signature handler to OEEquivHash
> >    testsuite: add support to run with bitbake hashserver
> >    isar-sstate: add support to clean hashes from hashequiv server
> >    rootfs: use host apt to install packages into rootfs
> >    bootstrap: do not include apt
> >    bootstrap: do not add ca-certificates
> > 
> >   RECIPE-API-CHANGELOG.md                       |  25 +++
> >   .../bitbake-user-manual-ref-variables.rst     |  23 +++
> >   bitbake/lib/bb/cooker.py                      |  11 +-
> >   bitbake/lib/bb/utils.py                       |   9 ++
> >   meta-isar/conf/distro/raspios-bookworm.conf   |   2 +
> >   meta-isar/conf/distro/raspios-bullseye.conf   |   2 +
> >   meta-isar/conf/distro/ubuntu-focal.conf       |   3 +
> >   .../optee-examples/files/debian/control.tmpl  |  26 ++--
> >   .../optee-examples-stm32mp15x_3.21.0.bb       |  36 ++---
> >   meta/classes-global/isar-events.bbclass       |   5 +
> >   meta/classes-recipe/container-loader.bbclass  |   1 +
> >   meta/classes-recipe/deb-dl-dir.bbclass        |  34 ++++
> >   meta/classes-recipe/debianize.bbclass         |   6 +-
> >   meta/classes-recipe/dpkg-raw.bbclass          |   3 +-
> >   meta/classes-recipe/dpkg-source.bbclass       |  79 ++--------
> >   meta/classes-recipe/dracut-module.bbclass     |   2 +-
> >   .../image-account-extension.bbclass           |   1 +
> >   meta/classes-recipe/linux-kernel.bbclass      |   3 +
> >   meta/classes-recipe/linux-module.bbclass      |   7 +-
> >   meta/classes-recipe/rootfs.bbclass            | 145 +++++++-----------
> >   meta/classes-recipe/sdk.bbclass               |   5 +-
> >   meta/conf/bitbake.conf                        |   3 +-
> >   .../optee-client/files/debian/control.tmpl    |   2 +-
> >   .../optee-ftpm/files/debian/control.tmpl      |   4 +-
> >   .../isar-mmdebstrap/isar-mmdebstrap.inc       |  44 +-----
> >   .../sbom-chroot/sbom-chroot.bb                |   2 +-
> >   .../sbuild-chroot/sbuild-chroot.inc           |   2 +
> >   .../linux-module/files/debian/control.tmpl    |   4 +-
> >   .../linux-module/files/debian/rules.tmpl      |   2 +-
> >   meta/recipes-kernel/linux/linux-distro.bb     |  12 +-
> >   scripts/isar-sstate                           |  39 ++++-
> >   testsuite/cibuilder.py                        |   2 +
> >   .../dockerdata/test-container-entrypoint      |   2 +-
> >   33 files changed, 291 insertions(+), 255 deletions(-)
> > 
> 
> Hi,
> 
> This series causes a failure in CI, and I can reproduce it consistently now.
> 
> Failing test is ReproTest.test_repro_signed, I ran it in rootful mode.

Hi, thanks for the log.

As we now have better caching and just generate the source package
once, we also just fetch once - which exposes underlying isar bugs.
These are actually independent of the chroot mode - but not independent
of the test execution order.

The main issue hereby is that currently base-apt bugs are shadowed by a
shared DL_DIR (or previously running tests), as base-apt simply imports
all packages from the DL_DIR. This is bad per-se, but not really worth
fixing as it is refactored in "Improving base-apt usage" anyways.

Now to the bug:

The apt source fetcher derives the DEBSRCDIR sub-directory from
BASE_DISTRO, which is resolved against OVERRIDES and therefore 
describes the recipe's PACKAGE_ARCH rather than the chroot the sources
are fetched in. On raspios, where BASE_DISTRO:arm64 is "raspios64", a
native recipe in mc:rpi-arm64-v8-bookworm caches its .dsc below
raspios-bookworm while base-apt only imports raspios64-bookworm and
debian-bookworm for that multiconfig. The sources are then missing from
the cached base repository and the offline rebuild of the repro test
fails in do_unpack.

I'll send the corresponding fixes as separate patches.

Felix

> 
> ERROR: Logfile of failure stored in: 
> /build/isar_wzh_devel_1z_others/72/build/tmp/work/raspios-bookworm-arm64/hello-native/1.0-r0/temp/log.do_unpack.1742588
> ERROR: Task 
> (mc:rpi-arm64-v8-bookworm:virtual:native:/build/isar_wzh_devel_1z_others/72/build/../meta-isar/recipes-app/hello/hello.bb:do_unpack) 
> failed with exit code '1'
> 
> Logs from the task:
> DEBUG: Executing python function do_unpack
> DEBUG: Executing shell function schroot_create_configs
> Creating 
> /etc/schroot/chroot.d/isar-jenkins-4889355d-e5f6-4ea3-80d6-d13c787ef5ad-1742588
> DEBUG: Shell function schroot_create_configs finished
> DEBUG: Running export PSEUDO_DISABLED=1; export 
> PATH="/build/isar_wzh_devel_1z_others/72/scripts:/build/isar_wzh_devel_1z_others/72/bitbake/bin:/tmp/avocado_venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin"; 
> export HOME="/var/lib/jenkins"; schroot -q -b -c 
> isar-jenkins-4889355d-e5f6-4ea3-80d6-d13c787ef5ad-1742588
> NOTE: Started session: 
> isar-jenkins-4889355d-e5f6-4ea3-80d6-d13c787ef5ad-1742588-b4abf875-2b4f-4df8-902f-3d8ce50e6b35
> DEBUG: Running export PSEUDO_DISABLED=1; export 
> PATH="/build/isar_wzh_devel_1z_others/72/scripts:/build/isar_wzh_devel_1z_others/72/bitbake/bin:/tmp/avocado_venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin"; 
> export HOME="/var/lib/jenkins";
>                  set -e
>                  schroot -r -c 
> isar-jenkins-4889355d-e5f6-4ea3-80d6-d13c787ef5ad-1742588-b4abf875-2b4f-4df8-902f-3d8ce50e6b35 
> -d / -u root --                     rm 
> /etc/apt/sources.list.d/isar-apt.list /etc/apt/preferences.d/isar-apt
>                  schroot -r -c 
> isar-jenkins-4889355d-e5f6-4ea3-80d6-d13c787ef5ad-1742588-b4abf875-2b4f-4df8-902f-3d8ce50e6b35 
> -d / --                     sh -c '
>                          set -e
>                          dscfile=$(apt-get -y -qq --print-uris 
> --only-source source hello |                                   cut -d " 
> " -f2 | grep -E "\.dsc")
>                          cp /downloads/deb-src/raspios-bookworm/hello/* 
> /home/builder/hello
>                          cd /home/builder/hello
>                          mv -f hello-native-1.0 hello-native-1.0.prev
>                          dpkg-source -x "$dscfile" hello-native-1.0
>                          find hello-native-1.0.prev -mindepth 1 
> -maxdepth 1 -exec mv {} hello-native-1.0/ \;
>                          rmdir hello-native-1.0.prev
>                          '
> 
> /build/isar_wzh_devel_1z_others/72/bitbake/lib/bb/data_smart.py:1034: 
> ResourceWarning: unclosed database in <sqlite3.Connection object at 
> 0x7f118e1c0e50>
>    klist = set()
> ResourceWarning: Enable tracemalloc to get the object allocation traceback
> DEBUG: Running export PSEUDO_DISABLED=1; export 
> PATH="/build/isar_wzh_devel_1z_others/72/scripts:/build/isar_wzh_devel_1z_others/72/bitbake/bin:/tmp/avocado_venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin"; 
> export HOME="/var/lib/jenkins"; schroot -q -f -e -c 
> isar-jenkins-4889355d-e5f6-4ea3-80d6-d13c787ef5ad-1742588-b4abf875-2b4f-4df8-902f-3d8ce50e6b35
> DEBUG: Executing shell function schroot_delete_configs
> Removing 
> /etc/schroot/isar-jenkins-4889355d-e5f6-4ea3-80d6-d13c787ef5ad-1742588
> Removing 
> /etc/schroot/chroot.d/isar-jenkins-4889355d-e5f6-4ea3-80d6-d13c787ef5ad-1742588
> DEBUG: Shell function schroot_delete_configs finished
> ERROR: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export 
> PATH="/build/isar_wzh_devel_1z_others/72/scripts:/build/isar_wzh_devel_1z_others/72/bitbake/bin:/tmp/avocado_venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin"; 
> export HOME="/var/lib/jenkins";
>                  set -e
>                  schroot -r -c 
> isar-jenkins-4889355d-e5f6-4ea3-80d6-d13c787ef5ad-1742588-b4abf875-2b4f-4df8-902f-3d8ce50e6b35 
> -d / -u root --                     rm 
> /etc/apt/sources.list.d/isar-apt.list /etc/apt/preferences.d/isar-apt
>                  schroot -r -c 
> isar-jenkins-4889355d-e5f6-4ea3-80d6-d13c787ef5ad-1742588-b4abf875-2b4f-4df8-902f-3d8ce50e6b35 
> -d / --                     sh -c '
>                          set -e
>                          dscfile=$(apt-get -y -qq --print-uris 
> --only-source source hello |                                   cut -d " 
> " -f2 | grep -E "\.dsc")
>                          cp /downloads/deb-src/raspios-bookworm/hello/* 
> /home/builder/hello
>                          cd /home/builder/hello
>                          mv -f hello-native-1.0 hello-native-1.0.prev
>                          dpkg-source -x "$dscfile" hello-native-1.0
>                          find hello-native-1.0.prev -mindepth 1 
> -maxdepth 1 -exec mv {} hello-native-1.0/ \;
>                          rmdir hello-native-1.0.prev
>                          '
>                   failed with exit code 1, output:
> E: You must put some 'deb-src' URIs in your sources.list
> 
> DEBUG: Python function do_unpack finished
> 
> Zhihang

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/9ec7540f9a0d4efffd53e8e452007fe43ef22ea4.camel%40siemens.com.