[PATCH v6 00/11] Improving base-apt usage

[Uladzimir Bely <ubely@ilbers.de>]

Currently, base-apt is used in the following way:

 - After the first build, all .deb files that took part in the process
are is cached in the ${DL_DIR}/deb/. All the packages are downloaded
from the remote repositories.

 - At the second build, `base-apt` repo is first created from the
previously downloaded packages (if ISAR_USE_CACHED_BASE_REPO is set).
Futher debootstrap and installing packages are done from this
local repo.

The idea of this patchset is to use local `base-apt` repo even during
the first build. Before running any task that requires additional
packages, these packages are predownloaded using `debrepo` script to
the `base-apt` repo and the debootstrapping / installation steps are
always done from it.

Such approach allows to stop using deb-del-dir import/export
functionality in favour of using local 'base-apt' repository.

New approach requires 'apt-get update' to be run after every call of
`debrepo` script. It differs from the old approach, when it was run
only once after debootstrapping and Debian package database considered
unchanged during the build.

But, potentially, new approach in combination with the patchset
"Introduce Debian dependencies and provides helper", could allow to
prepopulate 'base-apt' repo before running any bootstrap/build tasks,
even at the first "clean" build.

Currently, separate `debrepo` script is used for prefetching packages
to the local 'base-apt' repo. It requires python3-apt to be installed
on the build host. Potentially, the functionality it provides could be
directly integrated to the `debrepo.bbclass`.

Some examples of using `debrepo` script in standalone mode:

```
# Create local `/build/ba` repository sufficient to debootstrap Debian
# base system with `armhf` architecture:
debrepo --init --workdir=/build/dr --repodir=/build/ba --arch=armhf
# Add some packages to this repo (e.g., build deps for some recipe):
debrepo --workdir=/build/dr locales gnupg
# Add srcpackages for some package to the repo:
debrepo --workdir=/build/dr --srcmode tzdata
```

This patchset v6 can be considered as the first "non-POC" version since
it was heavily tested in CI and looks stable enough.

Known issues:
 - May stuck when 'BASE_REPO_FEATURES = "cache-deb-src"' set.

Changes since v5:
 - Rebased on latest next.
 - Changed order of the patches.
 - Fixes in `debrepo` script that allow to use it outside Isar in
'standalone' mode.

Changes since v4:
 - Rebased on latest next.
 - Rearranged patches since some of them are already in next.
 - Added possibility to select between new "prefetch" base-apt mode and
old behaviour when it's populated on 2nd build with packages downloaded
during 1st build. New behaviour is disabled by default, but enabled in
local.conf.example for testing purposes.
 - Code passes both full and fast CI in both "old" and "new" modes.

Changes since v3:
 - Rebased on latest next.
 - Cross-building for raspberry supported.
 - Code passes both full and fast CI.

Changes since v2:
 - Populate base-apt before using at all steps of native build.

Changes since v1:
 - Rebased on latest next.
 - Updated patchset description.

Uladzimir Bely (11):
  scripts: Add debrepo python script handling base-apt
  meta: Add debrepo bbclass handling base-apt prefetching
  meta: Always use base-apt repo in local mode
  meta: Use cached base-apt repo to debootstrap
  meta: Consider global debrepo context
  base-apt: Predownload packages to base-apt before install
  meta: Add cache-deb-src functionality in base-apt mode
  testsuite: Set ISAR_PREFETCH_BASE_APT by default
  Disable deb-dl-dir in base-apt prefetch mode
  kas: Add PREFETCH_BASE_APT config entry
  ci_build.sh: Install python3-apt if not installed

 RECIPE-API-CHANGELOG.md                       |   9 +
 kas/opt/Kconfig                               |  13 +
 kas/opt/prefetch-base-apt.yaml                |   9 +
 meta-test/conf/local.conf.sample              |   3 +
 meta/classes/deb-dl-dir.bbclass               |  19 +
 meta/classes/debrepo.bbclass                  |  81 +++
 meta/classes/dpkg-base.bbclass                |  38 +-
 meta/classes/dpkg.bbclass                     |   8 +
 meta/classes/image-locales-extension.bbclass  |   5 +
 meta/classes/image-tools-extension.bbclass    |   9 +
 meta/classes/rootfs.bbclass                   |  12 +-
 meta/conf/bitbake.conf                        |   5 +
 .../isar-bootstrap/isar-bootstrap.inc         | 106 +++-
 meta/recipes-devtools/base-apt/base-apt.bb    |  21 +-
 .../sbuild-chroot/sbuild-chroot.inc           |  11 +
 scripts/ci_build.sh                           |   8 +-
 scripts/debrepo                               | 571 ++++++++++++++++++
 testsuite/cibase.py                           |   4 +
 testsuite/cibuilder.py                        |   8 +-
 19 files changed, 916 insertions(+), 24 deletions(-)
 create mode 100644 kas/opt/prefetch-base-apt.yaml
 create mode 100644 meta/classes/debrepo.bbclass
 create mode 100755 scripts/debrepo

-- 
2.43.0