Re: [v11 00/12] Improving base-apt usage

["'MOESSBAUER, Felix' via isar-users" <isar-users@googlegroups.com>]

On Fri, 2026-10-02 at 09:01 +0200, akarpovich wrote:
> `base-apt` is a local apt repository containing all upstream (Debian, Raspberry
> Pi OS, Ubuntu...) packages needed for a particular build. This series implements
> upfront repository downloading. This is the first step towards local partial
> mirror management.
> 
> The current approach in `next`:
> 
> - On the first build, mmdebstrap and sbuild are used for building Isar
> artifacts. The packages downloaded from the Internet are cached in local
> directories.
> - On the next build:
>   - Analyze the logs from the previous build, save packages downloaded by the
>   bootstraps, sbuilds and imagers into `base-apt`.
>   - Use `base-apt` for bootstrapping, building and image creation.
> 
> Some issues with the current approach:
> 
> 1. Different policies must be followed for the first and the subsequent builds.
> 2. As we have multiple versions of the same package from the main and security
> repositories and rely on build logs and `find` for populating `base-apt`, extra
> care must be taken to ensure that the right package version lands in `base-apt`.
> 3. We rely on internal implementation of `mmdebstrap` and `sbuild` for saving
> and reusing the packages.
> 4. Source packages are stored in a different flat directory, `apt-get source`
> for upstream packages is not possible.
> 5. At the moment of `base-apt` creation all we have is the package name. The
> knowledge about the upstream repositories is lost and no local repository
> policy implementation is possible (e.g., for the "multiple products, multiple
> distros" use case).
> 6. For implementing further use cases like "fetch all sources necessary for
> bootstrapping the base system itself", additional logic is necessary.
> 
> The new approach:
> 
> - On the first build:
>   - All packages necessary for bootstrapping and building are identified and
>   downloaded upfront.
>   - `base-apt` is used for bootstrapping, building and image creation.
> - On the next build:
>   - `base-apt` is used for bootstrapping, building and image creation.
> 
> This series addresses issues 1-5 and provides the architecture for
> implementing further use cases.
> 
> The new approach is enabled by default. Setting `ISAR_PREFETCH_BASE_APT` to
> zero falls back to the old approach.

Hi,

I'm really wondering if we should rather drop the old approach. I'm not
a base-apt user, so I can't say much. However the existing base-apt
code is quite complex - which I learned while doing the host-apt parts.
Now we add yet another implementation that is also non trivial.

Please also check this series against the "More build time reductions"
patch series which switches to use the host apt in most cases
(especially for the image rootfs assembly, so we don't need apt in the
rootfs).

> 
> The implementation uses `debrepo` script which can also be called manually for
> pre-fetching packages to the local 'base-apt' repository. It requires
> `python3-apt` to be installed on the build host. Some examples of its usage in
> standalone mode:
> 
> ```
> # Create local `/build/ba` repository sufficient to bootstrap Debian system with
> `armhf` architecture:
> debrepo --init --workdir=/build/dr --repodir=/build/ba --arch=armhf
> 
> # Add some packages to this repo (e.g., build deps for some recipe):
> debrepo --workdir=/build/dr locales gnupg
> 
> # Add srcpackages for some package to the repo:
> debrepo --workdir=/build/dr --srcmode tzdata
> ```
> 
> Changes since v10:
>  - Rebased on fresh next (1d8791f5)
>  - Added wget in install section of user manual
> 
> Changes since v9:
>  - Rebased on fresh next (436bc96f)
>  - Removed "Fix python3-apt dependency"
> 
> Changes since v8:
>  - Rebased on fresh next (86c761e4)
>  - Removed fixes for isar-apt download in rootfs (use Felix's fix)
>  - Renamed debrepo -> deb-repo
>  - Fixed failed tests on Full and Fast CI testsets
> 
> 
> Aliaksei Karpovich (2):
>   debrepo: Add override for reprepro
>   base-apt: Move bullseye-mipsel in separate dir
> 
> Uladzimir Bely (10):
>   scripts: Add debrepo python script handling base-apt
>   meta: Add debrepo bbclass handling base-apt prefetching
>   meta: Always use base-apt repo in local mode
>   meta: Use cached base-apt repo to bootstrap
>   base-apt: Predownload packages to base-apt before install
>   meta: Add cache-deb-src functionality in base-apt mode
>   testsuite: Set ISAR_PREFETCH_BASE_APT by default
>   Disable deb-dl-dir in base-apt prefetch mode
>   kas: Add PREFETCH_BASE_APT config entry
>   ci_build.sh: Install python3-apt if not installed
> 
>  RECIPE-API-CHANGELOG.md                       |  10 +
>  doc/user_manual.md                            |   2 +
>  kas/opt/Kconfig                               |  12 +
>  kas/opt/prefetch-base-apt.yaml                |   9 +
>  meta-test/conf/local.conf.sample              |   3 +
>  meta/classes-recipe/bootstrap.bbclass         |  50 +-
>  meta/classes-recipe/crossvars.bbclass         |   1 +
>  meta/classes-recipe/deb-dl-dir.bbclass        |  43 ++
>  meta/classes-recipe/dpkg-base.bbclass         |   1 +
>  meta/classes-recipe/dpkg.bbclass              |   8 +
>  .../image-locales-extension.bbclass.orig      | 159 ++++

This file should not be included into the series.

Felix

>  .../image-tools-extension.bbclass             |  13 +
>  meta/classes-recipe/rootfs.bbclass            |  15 +-
>  meta/classes-recipe/sbuild.bbclass            |   3 +-
>  meta/classes/debrepo.bbclass                  |  89 +++
>  meta/conf/bitbake.conf                        |  10 +-
>  meta/conf/distro/debian-bullseye.conf         |   1 +
>  meta/lib/aptsrc_fetcher.py                    |  46 +-
>  .../isar-mmdebstrap/isar-mmdebstrap-host.bb   |   2 +
>  .../isar-mmdebstrap/isar-mmdebstrap.inc       |  22 +-
>  meta/recipes-devtools/base-apt/base-apt.bb    |  21 +-
>  .../sbom-chroot/sbom-chroot.bb                |   2 +
>  .../sbuild-chroot/sbuild-chroot-host.bb       |   2 +
>  scripts/ci_build.sh                           |   8 +-
>  scripts/deb-repo                              | 689 ++++++++++++++++++
>  testsuite/cibase.py                           |   4 +
>  testsuite/cibuilder.py                        |   7 +
>  27 files changed, 1203 insertions(+), 29 deletions(-)
>  create mode 100644 kas/opt/prefetch-base-apt.yaml
>  create mode 100644 meta/classes-recipe/image-locales-extension.bbclass.orig
>  create mode 100644 meta/classes/debrepo.bbclass
>  create mode 100755 scripts/deb-repo
> 
> --
> 2.53.0
> 
> --
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/isar-users/20261002070335.26526-1-akarpovich%40ilbers.de.

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/9a38880d7f2195d5fc5a1940848156d35745ffba.camel%40siemens.com.