[PATCH v4 0/4] non-privileged commands in chroot

["'Cedric Hombourger' via isar-users" <isar-users@googlegroups.com>]

When building root filesystems for foreign architectures with package source
caching enabled, apt operations are executed within the rootfs through QEMU
emulation. This results in significantly degraded performance, particularly
when downloading source packages sequentially.

This patch series introduces a new wrapper function that enables native
command execution against a rootfs while preserving special mount points
(such as /isar-apt). The approach:

- Improves build performance for foreign architecture builds
- Maintains filesystem isolation using bubblewrap
- Preserves access to special mount points required by isar

Additional notes:
- rootfs_cmd may be used to run commands from the host root file-system:
  use with extreme care to avoid host contamination problems.
- mmdebstrap already calls apt-get of the host to download packages (in
  other words, a build of a bookworm image from a trixie host will
  result in mmdebstrap (from trixie), call apt-get (from trixie) to
  download bookworm packages. This is the behavior we have before and
  after these changes.
- With these changes and when caching of Debian source packages is
  enabled/requested, Isar will use apt-get of the host to download
  source packages (it will however do this from a bubblewrap'ed
  environment to avoid a non-required privilege elevation; Isar has
  many but we need to start from somewhere).

Testing:
- Tested against 9e62337953fbb8371c846c44e8a99d62a8d220ba
- Basic smoke tests performed successfully (citest.py -t fast)
- Performance improvements observed in source package acquisition
- Tested with various foreign architecture configurations

Dependencies:
- Adds bubblewrap as a new host tool requirement
- Uses kas-container 4.8.0 or later (see [1])

Changes since v3 patch:
  - drop image-postproc-extension patches (refactoring and use of
    rootfs_cmd). They are not strictly needed and were only meant
    to provide another potential use of rootfs_cmd.
  - Rebase changes to RECIPE-API-CHANGELOG.md and added a few extra
    words about the motivation.

Changes since v2 patch:
  - rootfs_install_pkgs_download will no longer use sudo to run
    apt-get install --download-only. This was added to further
    demonstrate/test rootfs_cmd in existing Isar code.

Changes since v1 patch:
  - Rebase (resolve RECIPE-API-CHANGELOG.md merge conflicts)
  - Prefix rootfs variable in rootfs_cmd with bwrap to avoid clashes

Changes since RFC patch:
  - Let caller decide where to bind-mount the rootfs to
  - Make the rootfs argument optional
  - Support 32-bit rootfs (no lib64 there)

Test Results (avocado started from a kas-container version 4.8.1):
 (01/22) citest.py:DevTest.test_dev: STARTED
 (01/22) citest.py:DevTest.test_dev: PASS (1132.17 s)
 (02/22) citest.py:DevTest.test_dev_apps: STARTED
 (02/22) citest.py:DevTest.test_dev_apps: PASS (845.24 s)
 (03/22) citest.py:DevTest.test_dev_rebuild: STARTED
 (03/22) citest.py:DevTest.test_dev_rebuild: PASS (689.53 s)
 (04/22) citest.py:DevTest.test_dev_run_amd64_bookworm: STARTED
 (04/22) citest.py:DevTest.test_dev_run_amd64_bookworm: PASS (53.79 s)
 (05/22) citest.py:DevTest.test_dev_run_arm64_bookworm: STARTED
 (05/22) citest.py:DevTest.test_dev_run_arm64_bookworm: PASS (32.64 s)
 (06/22) citest.py:DevTest.test_dev_run_arm_bookworm: STARTED
 (06/22) citest.py:DevTest.test_dev_run_arm_bookworm: PASS (34.15 s)
 (07/22) citest.py:CrossTest.test_cross: STARTED
 (07/22) citest.py:CrossTest.test_cross: PASS (488.24 s)
 (08/22) citest.py:CrossTest.test_cross_debsrc: STARTED
 (08/22) citest.py:CrossTest.test_cross_debsrc: PASS (1409.06 s)
 (09/22) citest.py:CrossTest.test_cross_trixie: STARTED
 (09/22) citest.py:CrossTest.test_cross_trixie: PASS (216.54 s)
 (10/22) citest.py:CrossTest.test_cross_kselftest: STARTED
 (10/22) citest.py:CrossTest.test_cross_kselftest: PASS (340.48 s)
 (11/22) citest.py:CrossTest.test_cross_rpi: STARTED
 (11/22) citest.py:CrossTest.test_cross_rpi: PASS (1053.48 s)
 (12/22) citest.py:VmBootTestFast.test_arm_bullseye: STARTED
 (12/22) citest.py:VmBootTestFast.test_arm_bullseye: PASS (41.03 s)
 (13/22) citest.py:VmBootTestFast.test_arm_bullseye_example_module: STARTED
 (13/22) citest.py:VmBootTestFast.test_arm_bullseye_example_module: PASS (7.07 s)
 (14/22) citest.py:VmBootTestFast.test_arm_bullseye_getty_target: STARTED
 (14/22) citest.py:VmBootTestFast.test_arm_bullseye_getty_target: PASS (7.82 s)
 (15/22) citest.py:VmBootTestFast.test_arm_buster: STARTED
 (15/22) citest.py:VmBootTestFast.test_arm_buster: PASS (37.54 s)
 (16/22) citest.py:VmBootTestFast.test_arm_buster_getty_target: STARTED
 (16/22) citest.py:VmBootTestFast.test_arm_buster_getty_target: PASS (6.79 s)
 (17/22) citest.py:VmBootTestFast.test_arm_buster_example_module: STARTED
 (17/22) citest.py:VmBootTestFast.test_arm_buster_example_module: PASS (7.57 s)
 (18/22) citest.py:VmBootTestFast.test_arm_bookworm: STARTED
 (18/22) citest.py:VmBootTestFast.test_arm_bookworm: PASS (49.58 s)
 (19/22) citest.py:VmBootTestFast.test_arm_bookworm_example_module: STARTED
 (19/22) citest.py:VmBootTestFast.test_arm_bookworm_example_module: PASS (8.06 s)
 (20/22) citest.py:VmBootTestFast.test_arm_bookworm_getty_target: STARTED
 (20/22) citest.py:VmBootTestFast.test_arm_bookworm_getty_target: PASS (8.18 s)
 (21/22) citest.py:VmBootTestFast.test_amd64_trixie: STARTED
 (21/22) citest.py:VmBootTestFast.test_amd64_trixie: PASS (37.14 s)
 (22/22) citest.py:VmBootTestFast.test_arm64_trixie: STARTED
 (22/22) citest.py:VmBootTestFast.test_arm64_trixie: PASS (41.79 s)
 RESULTS    : PASS 22 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
 JOB TIME   : 6585.87 s 

cedric.hombourger@siemens.com (4):
  rootfs: introduce wrapper to run commands against a rootfs
  deb-dl-dir: optimize caching of source packages using apt natively
  bootstrap: create lock for downloads/deb without sudo
  rootfs: do not get elevated privileges when downloading packages

 RECIPE-API-CHANGELOG.md                       |  8 ++
 doc/user_manual.md                            |  1 +
 meta/classes/deb-dl-dir.bbclass               | 58 ++++++-------
 meta/classes/rootfs.bbclass                   | 83 ++++++++++++++++++-
 .../isar-mmdebstrap/isar-mmdebstrap.inc       |  4 +
 5 files changed, 120 insertions(+), 34 deletions(-)

-- 
2.47.3

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250925065433.4180883-1-cedric.hombourger%40siemens.com.