[PATCH v4 0/4] non-privileged commands in chroot

[PATCH v4 0/4] non-privileged commands in chroot

['Cedric Hombourger' via isar-users]

When building root filesystems for foreign architectures with package source
caching enabled, apt operations are executed within the rootfs through QEMU
emulation. This results in significantly degraded performance, particularly
when downloading source packages sequentially.

This patch series introduces a new wrapper function that enables native
command execution against a rootfs while preserving special mount points
(such as /isar-apt). The approach:

- Improves build performance for foreign architecture builds
- Maintains filesystem isolation using bubblewrap
- Preserves access to special mount points required by isar

Additional notes:
- rootfs_cmd may be used to run commands from the host root file-system:
  use with extreme care to avoid host contamination problems.
- mmdebstrap already calls apt-get of the host to download packages (in
  other words, a build of a bookworm image from a trixie host will
  result in mmdebstrap (from trixie), call apt-get (from trixie) to
  download bookworm packages. This is the behavior we have before and
  after these changes.
- With these changes and when caching of Debian source packages is
  enabled/requested, Isar will use apt-get of the host to download
  source packages (it will however do this from a bubblewrap'ed
  environment to avoid a non-required privilege elevation; Isar has
  many but we need to start from somewhere).

Testing:
- Tested against 9e62337953fbb8371c846c44e8a99d62a8d220ba
- Basic smoke tests performed successfully (citest.py -t fast)
- Performance improvements observed in source package acquisition
- Tested with various foreign architecture configurations

Dependencies:
- Adds bubblewrap as a new host tool requirement
- Uses kas-container 4.8.0 or later (see [1])

Changes since v3 patch:
  - drop image-postproc-extension patches (refactoring and use of
    rootfs_cmd). They are not strictly needed and were only meant
    to provide another potential use of rootfs_cmd.
  - Rebase changes to RECIPE-API-CHANGELOG.md and added a few extra
    words about the motivation.

Changes since v2 patch:
  - rootfs_install_pkgs_download will no longer use sudo to run
    apt-get install --download-only. This was added to further
    demonstrate/test rootfs_cmd in existing Isar code.

Changes since v1 patch:
  - Rebase (resolve RECIPE-API-CHANGELOG.md merge conflicts)
  - Prefix rootfs variable in rootfs_cmd with bwrap to avoid clashes

Changes since RFC patch:
  - Let caller decide where to bind-mount the rootfs to
  - Make the rootfs argument optional
  - Support 32-bit rootfs (no lib64 there)

Test Results (avocado started from a kas-container version 4.8.1):
 (01/22) citest.py:DevTest.test_dev: STARTED
 (01/22) citest.py:DevTest.test_dev: PASS (1132.17 s)
 (02/22) citest.py:DevTest.test_dev_apps: STARTED
 (02/22) citest.py:DevTest.test_dev_apps: PASS (845.24 s)
 (03/22) citest.py:DevTest.test_dev_rebuild: STARTED
 (03/22) citest.py:DevTest.test_dev_rebuild: PASS (689.53 s)
 (04/22) citest.py:DevTest.test_dev_run_amd64_bookworm: STARTED
 (04/22) citest.py:DevTest.test_dev_run_amd64_bookworm: PASS (53.79 s)
 (05/22) citest.py:DevTest.test_dev_run_arm64_bookworm: STARTED
 (05/22) citest.py:DevTest.test_dev_run_arm64_bookworm: PASS (32.64 s)
 (06/22) citest.py:DevTest.test_dev_run_arm_bookworm: STARTED
 (06/22) citest.py:DevTest.test_dev_run_arm_bookworm: PASS (34.15 s)
 (07/22) citest.py:CrossTest.test_cross: STARTED
 (07/22) citest.py:CrossTest.test_cross: PASS (488.24 s)
 (08/22) citest.py:CrossTest.test_cross_debsrc: STARTED
 (08/22) citest.py:CrossTest.test_cross_debsrc: PASS (1409.06 s)
 (09/22) citest.py:CrossTest.test_cross_trixie: STARTED
 (09/22) citest.py:CrossTest.test_cross_trixie: PASS (216.54 s)
 (10/22) citest.py:CrossTest.test_cross_kselftest: STARTED
 (10/22) citest.py:CrossTest.test_cross_kselftest: PASS (340.48 s)
 (11/22) citest.py:CrossTest.test_cross_rpi: STARTED
 (11/22) citest.py:CrossTest.test_cross_rpi: PASS (1053.48 s)
 (12/22) citest.py:VmBootTestFast.test_arm_bullseye: STARTED
 (12/22) citest.py:VmBootTestFast.test_arm_bullseye: PASS (41.03 s)
 (13/22) citest.py:VmBootTestFast.test_arm_bullseye_example_module: STARTED
 (13/22) citest.py:VmBootTestFast.test_arm_bullseye_example_module: PASS (7.07 s)
 (14/22) citest.py:VmBootTestFast.test_arm_bullseye_getty_target: STARTED
 (14/22) citest.py:VmBootTestFast.test_arm_bullseye_getty_target: PASS (7.82 s)
 (15/22) citest.py:VmBootTestFast.test_arm_buster: STARTED
 (15/22) citest.py:VmBootTestFast.test_arm_buster: PASS (37.54 s)
 (16/22) citest.py:VmBootTestFast.test_arm_buster_getty_target: STARTED
 (16/22) citest.py:VmBootTestFast.test_arm_buster_getty_target: PASS (6.79 s)
 (17/22) citest.py:VmBootTestFast.test_arm_buster_example_module: STARTED
 (17/22) citest.py:VmBootTestFast.test_arm_buster_example_module: PASS (7.57 s)
 (18/22) citest.py:VmBootTestFast.test_arm_bookworm: STARTED
 (18/22) citest.py:VmBootTestFast.test_arm_bookworm: PASS (49.58 s)
 (19/22) citest.py:VmBootTestFast.test_arm_bookworm_example_module: STARTED
 (19/22) citest.py:VmBootTestFast.test_arm_bookworm_example_module: PASS (8.06 s)
 (20/22) citest.py:VmBootTestFast.test_arm_bookworm_getty_target: STARTED
 (20/22) citest.py:VmBootTestFast.test_arm_bookworm_getty_target: PASS (8.18 s)
 (21/22) citest.py:VmBootTestFast.test_amd64_trixie: STARTED
 (21/22) citest.py:VmBootTestFast.test_amd64_trixie: PASS (37.14 s)
 (22/22) citest.py:VmBootTestFast.test_arm64_trixie: STARTED
 (22/22) citest.py:VmBootTestFast.test_arm64_trixie: PASS (41.79 s)
 RESULTS    : PASS 22 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
 JOB TIME   : 6585.87 s 

cedric.hombourger@siemens.com (4):
  rootfs: introduce wrapper to run commands against a rootfs
  deb-dl-dir: optimize caching of source packages using apt natively
  bootstrap: create lock for downloads/deb without sudo
  rootfs: do not get elevated privileges when downloading packages

 RECIPE-API-CHANGELOG.md                       |  8 ++
 doc/user_manual.md                            |  1 +
 meta/classes/deb-dl-dir.bbclass               | 58 ++++++-------
 meta/classes/rootfs.bbclass                   | 83 ++++++++++++++++++-
 .../isar-mmdebstrap/isar-mmdebstrap.inc       |  4 +
 5 files changed, 120 insertions(+), 34 deletions(-)

-- 
2.47.3

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250925065433.4180883-1-cedric.hombourger%40siemens.com.

[PATCH v4 1/4] rootfs: introduce wrapper to run commands against a rootfs

['Cedric Hombourger' via isar-users]

From: "cedric.hombourger@siemens.com" <cedric.hombourger@siemens.com>

"sudo chroot" is used in several places to run commands inside rootfs
directories constructed by Isar. There are cases where a command could
be used without elevated privileges as long as special folders such as
/isar-apt are mounted (they are often referenced as /isar-apt in
configuration files found in the target rootfs). For such cases,
bubblewrap may be used to create a non-privileged namespace (either
in a bare/native environment or within a docker/podman container)
where the command will be executed as if chroot had been used. The
rootfs may also be the host root file-system: this should however
be used with care to avoid host contamination problems (note: Isar
already relies on a number of host tools).

Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
---
 RECIPE-API-CHANGELOG.md     |  8 +++++
 doc/user_manual.md          |  1 +
 meta/classes/rootfs.bbclass | 67 +++++++++++++++++++++++++++++++++++++
 3 files changed, 76 insertions(+)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 92e7811c..53e650d4 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -741,3 +741,11 @@ By setting `MS_TPM_20_REF_DIR` in an optee-ftpm recipe, it is now possible to
 use the new optee_ftpm code base from the OP-TEE project. That variable has to
 point to a subdir in `WORKDIR` which contains the unpacked ms-tpm-20-ref source
 code.
+
+### Require bubblewrap to run non-privileged commands with bind-mounts
+
+Isar occasionally needs to run commands within root file-systems that it
+builds and with several bind-mounts (e.g. /isar-apt). bubblewrap may be
+used in Isar classes instead of `sudo chroot` to avoid unecessary privilege
+elevations (when we "just" need to chroot but do not require root). It is
+pre-installed in kas-container version 4.8 (or later).
diff --git a/doc/user_manual.md b/doc/user_manual.md
index 67f91973..be89ce1d 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -75,6 +75,7 @@ Install the following packages:
 ```
 apt install \
   binfmt-support \
+  bubblewrap \
   bzip2 \
   mmdebstrap \
   arch-test \
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index ebe3bf4a..f740c6e1 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -34,6 +34,73 @@ export LANG = "C"
 export LANGUAGE = "C"
 export LC_ALL = "C"
 
+# Execute a command against a rootfs and with isar-apt bind-mounted.
+# Additional mounts may be specified using --bind <source> <target> and a
+# custom directory for the command to be executed with --chdir <dir>. The
+# command is assumed to follow the special "--" argument. This would replace
+# "sudo chroot" calls especially when a native command may be used instead of
+# chroot'ed command and without elevated privileges (the command will likely
+# take the rootfs as argument; e.g. apt-get -o Dir=${ROOTFSDIR}). If the
+# optional rootfs argument is omitted, the host rootfs will be used (e.g. to
+# run native commands): this should be used with care.
+#
+# Usage: rootfs_cmd [options] [rootfs] -- command
+#
+rootfs_cmd() {
+    set -- "$@"
+    bwrap_args="--bind ${REPO_ISAR_DIR}/${DISTRO} /isar-apt"
+    bwrap_binds=""
+    bwrap_rootfs=""
+
+    while [ "${#}" -gt "0" ] && [ "${1}" != "--" ]; do
+        case "${1}" in
+            --bind)
+                if [ "${#}" -lt "3" ]; then
+                    bbfatal "--bind requires two arguments"
+                fi
+                bwrap_binds="${bwrap_binds} --bind ${2} ${3}"
+                shift 3
+                ;;
+            --chdir)
+                if [ "${#}" -lt "2" ]; then
+                    bbfatal "${1} requires an argument"
+                fi
+                bwrap_args="${bwrap_args} ${1} ${2}"
+                shift 2
+                ;;
+            -*)
+                bbfatal "${1} is not a supported option!"
+                ;;
+            *)
+                if [ -z "${bwrap_rootfs}" ]; then
+                    bwrap_rootfs="${1}"
+                    shift
+                else
+                    bbfatal "unexpected argument '${1}'"
+                fi
+                ;;
+        esac
+    done
+
+    if [ -n "${bwrap_rootfs}" ]; then
+        bwrap_args="${bwrap_args} --bind ${bwrap_rootfs} /"
+    fi
+
+    if [ "${#}" -le "1" ] || [ "${1}" != "--" ]; then
+        bbfatal "no command specified (missing --)"
+    fi
+    shift  # remove "--", command and its arguments follows
+
+    for ro_d in bin etc lib lib64 sys usr var; do
+        [ -d ${bwrap_rootfs}/${ro_d} ] || continue
+        bwrap_args="${bwrap_args} --ro-bind ${bwrap_rootfs}/${ro_d} /${ro_d}"
+    done
+
+    bwrap --unshare-user --unshare-pid ${bwrap_args} \
+        --dev-bind /dev /dev --proc /proc --tmpfs /tmp \
+        ${bwrap_binds} -- "${@}"
+}
+
 rootfs_do_mounts[weight] = "3"
 rootfs_do_mounts() {
     sudo -s <<'EOSUDO'
-- 
2.47.3

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250925065433.4180883-2-cedric.hombourger%40siemens.com.

[PATCH v4 2/4] deb-dl-dir: optimize caching of source packages using apt natively

['Cedric Hombourger' via isar-users]

From: "cedric.hombourger@siemens.com" <cedric.hombourger@siemens.com>

source package are downloaded by entering the target rootfs and run
apt there. For foreign architectures, this results in apt being
executed under QEMU and leads to poor performance. By using the
recently introduced rootfs_native_cmd command wrapper, apt will be
executed natively against the target rootfs and without elevated
privileges. For our test work-load, caching was reduced from more
than 10 hours to an hour. Performance is also more consistent as
it will no longer depend as to when bitbake kicks caching of
source packages for foreign architecture rootfs vs rootfs for the
host (in multiconfig builds).

Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
---
 meta/classes/deb-dl-dir.bbclass | 37 ++++++++-------------------------
 1 file changed, 9 insertions(+), 28 deletions(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 4780be20..ea0ed3d2 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -5,25 +5,6 @@
 
 inherit repository
 
-debsrc_do_mounts() {
-    sudo -s <<EOSUDO
-    set -e
-    mkdir -p "${1}/deb-src"
-    mountpoint -q "${1}/deb-src" || \
-    mount -o bind,private "${DEBSRCDIR}" "${1}/deb-src"
-EOSUDO
-}
-
-debsrc_undo_mounts() {
-    sudo -s <<EOSUDO
-    set -e
-    mkdir -p "${1}/deb-src"
-    mountpoint -q "${1}/deb-src" && \
-    umount "${1}/deb-src"
-    rm -rf "${1}/deb-src"
-EOSUDO
-}
-
 debsrc_source_version_filter() {
     # Filter the input to only consider Package, Version and Source lines
     #
@@ -51,11 +32,6 @@ debsrc_download() {
     export rootfs_distro="$2"
     mkdir -p "${DEBSRCDIR}"/"${rootfs_distro}"
 
-    debsrc_do_mounts "${rootfs}"
-
-    trap 'exit 1' INT HUP QUIT TERM ALRM USR1
-    trap 'debsrc_undo_mounts "${rootfs}"' EXIT
-
     ( flock 9
     set -e
     printenv | grep -q BB_VERBOSE_LOGS && set -x
@@ -96,13 +72,18 @@ debsrc_download() {
         dscname="${src}_${version#*:}.dsc"
         [ -f "${DEBSRCDIR}"/"${rootfs_distro}"/"${src}"/"${dscname}" ] || {
             # use apt-get source to download sources in DEBSRCDIR
-            sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
-                sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"
+            mkdir -p "${DEBSRCDIR}/${rootfs_distro}"/"${src}"
+            rootfs_cmd \
+                --bind "${DEBSRCDIR}" "/deb-src" \
+                --bind "${rootfs}" "${rootfs}" \
+                --chdir "/deb-src/${rootfs_distro}/${src}" \
+                -- \
+                apt-get -o APT::Architecture=${DISTRO_ARCH} \
+                        -o Dir="${rootfs}" -y --download-only \
+                        --only-source source "${src}=${version}"
         }
     done
     ) 9>"${DEBSRCDIR}/${rootfs_distro}.lock"
-
-    debsrc_undo_mounts "${rootfs}"
 }
 
 dbg_pkgs_download() {
-- 
2.47.3

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250925065433.4180883-3-cedric.hombourger%40siemens.com.

[PATCH v4 3/4] bootstrap: create lock for downloads/deb without sudo

['Cedric Hombourger' via isar-users]

From: "cedric.hombourger@siemens.com" <cedric.hombourger@siemens.com>

The syncin/syncout commands passed to mmdebstrap will create a lock
file in downloads/deb if it does not exist. As mmdebstrap is being
executed as root, the lock would also be owned by root and this will
cause problems for rootless commands that may be executed later (such
as downloading of Debian packages). Create the lock file without
sudo prior to running mmdebstrap for it to be owned by the build user
rather than root.

Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
---
 meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
index 931f6f13..b2de61ad 100644
--- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
+++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
@@ -181,6 +181,10 @@ do_bootstrap() {
                                  && sudo umount $base_apt_tmp \
                                  && rm -rf --one-file-system $base_apt_tmp' EXIT
 
+    # Create lock file so that it is owned by the user running the build (not root)
+    mkdir -p ${DEBDIR}
+    touch ${DEB_DL_LOCK}
+
     sudo TMPDIR="${BOOTSTRAP_TMPDIR}" mmdebstrap $bootstrap_args \
                    $arch_param \
                    --mode=unshare \
-- 
2.47.3

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250925065433.4180883-4-cedric.hombourger%40siemens.com.

[PATCH v4 4/4] rootfs: do not get elevated privileges when downloading packages

['Cedric Hombourger' via isar-users]

From: "cedric.hombourger@siemens.com" <cedric.hombourger@siemens.com>

Use rootfs_cmd() to run "apt-get install --download-only" without sudo.
This requires /var/cache/apt/archives/ to be writable by the build
user: change ownership while populating that folder with previously
downloaded packages (those in downloads/deb/).

Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
---
 meta/classes/deb-dl-dir.bbclass | 21 ++++++++++++++++++---
 meta/classes/rootfs.bbclass     | 16 +++++++++++++---
 2 files changed, 31 insertions(+), 6 deletions(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index ea0ed3d2..16ccd426 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -107,9 +107,24 @@ dbg_pkgs_download() {
 deb_dl_dir_import() {
     export pc="${DEBDIR}/${2}"
     export rootfs="${1}"
-    sudo mkdir -p "${rootfs}"/var/cache/apt/archives/
+    export uid=$(id -u)
+    export gid=$(id -g)
+
+    # let our unprivileged user place downloaded packages in /var/cache/apt/archives/
+    sudo -Es << '    EOSUDO'
+        mkdir -p "${rootfs}"/var/cache/apt/archives/partial/
+        touch "${rootfs}"/var/cache/apt/archives/lock
+        chown -R ${uid}:${gid} "${rootfs}"/var/cache/apt/archives/
+    EOSUDO
+
+    # nothing to copy if download directory does not exist just yet
     [ ! -d "${pc}" ] && return 0
-    flock -s "${pc}".lock sudo -Es << 'EOSUDO'
+
+    # attempt to create hard-links for .deb files from downloads/ into
+    # /var/cache/apt/archives/ so apt will only download packages we
+    # have not yet downloaded. perform a regular copy whenever hard-links
+    # cannot be created
+    ( flock 9
         set -e
         printenv | grep -q BB_VERBOSE_LOGS && set -x
 
@@ -118,7 +133,7 @@ deb_dl_dir_import() {
             ln -Pf -t "${rootfs}"/var/cache/apt/archives/ "$p" 2>/dev/null ||
                 cp -n --no-preserve=owner -t "${rootfs}"/var/cache/apt/archives/ "$p"
         done
-EOSUDO
+    ) 9>"${pc}".lock
 }
 
 deb_dl_dir_export() {
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index f740c6e1..684d04c4 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -286,10 +286,20 @@ ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_download"
 rootfs_install_pkgs_download[weight] = "600"
 rootfs_install_pkgs_download[progress] = "custom:rootfs_progress.PkgsDownloadProgressHandler"
 rootfs_install_pkgs_download[isar-apt-lock] = "release-after"
-rootfs_install_pkgs_download[network] = "${TASK_USE_NETWORK_AND_SUDO}"
+rootfs_install_pkgs_download[network] = "${TASK_USE_NETWORK}"
 rootfs_install_pkgs_download() {
-    sudo -E chroot '${ROOTFSDIR}' \
-        /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only ${ROOTFS_PACKAGES}
+    mkdir -p "${WORKDIR}/dpkg"
+
+    # Use our own dpkg lock files rather than those in the rootfs since we are not root
+    # (this is safe as there are no concurrent apt/dpkg operations for that rootfs)
+    touch "${WORKDIR}/dpkg/lock" "${WORKDIR}/dpkg/lock-frontend"
+
+    # download packages using apt in a non-privileged namespace
+    rootfs_cmd --bind "${ROOTFSDIR}/var/cache/apt/archives" /var/cache/apt/archives \
+               --bind "${WORKDIR}/dpkg/lock" /var/lib/dpkg/lock \
+               --bind "${WORKDIR}/dpkg/lock-frontend" /var/lib/dpkg/lock-frontend \
+               ${ROOTFSDIR} \
+               -- /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only ${ROOTFS_PACKAGES}
 }
 
 ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT ??= ""
-- 
2.47.3

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250925065433.4180883-5-cedric.hombourger%40siemens.com.

Re: [PATCH v4 2/4] deb-dl-dir: optimize caching of source packages using apt natively

['MOESSBAUER, Felix' via isar-users]

On Thu, 2025-09-25 at 08:54 +0200, 'Cedric Hombourger' via isar-users
wrote:
> From: "cedric.hombourger@siemens.com" <cedric.hombourger@siemens.com>
> 
> source package are downloaded by entering the target rootfs and run
> apt there. For foreign architectures, this results in apt being
> executed under QEMU and leads to poor performance. By using the
> recently introduced rootfs_native_cmd command wrapper, apt will be
> executed natively against the target rootfs and without elevated
> privileges. For our test work-load, caching was reduced from more
> than 10 hours to an hour. Performance is also more consistent as
> it will no longer depend as to when bitbake kicks caching of
> source packages for foreign architecture rootfs vs rootfs for the
> host (in multiconfig builds).
> 
> Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
> ---
>  meta/classes/deb-dl-dir.bbclass | 37 ++++++++-------------------------
>  1 file changed, 9 insertions(+), 28 deletions(-)
> 
> diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
> index 4780be20..ea0ed3d2 100644
> --- a/meta/classes/deb-dl-dir.bbclass
> +++ b/meta/classes/deb-dl-dir.bbclass
> @@ -5,25 +5,6 @@
>  
>  inherit repository
>  
> -debsrc_do_mounts() {
> -    sudo -s <<EOSUDO
> -    set -e
> -    mkdir -p "${1}/deb-src"
> -    mountpoint -q "${1}/deb-src" || \
> -    mount -o bind,private "${DEBSRCDIR}" "${1}/deb-src"
> -EOSUDO
> -}
> -
> -debsrc_undo_mounts() {
> -    sudo -s <<EOSUDO
> -    set -e
> -    mkdir -p "${1}/deb-src"
> -    mountpoint -q "${1}/deb-src" && \
> -    umount "${1}/deb-src"
> -    rm -rf "${1}/deb-src"
> -EOSUDO
> -}
> -
>  debsrc_source_version_filter() {
>      # Filter the input to only consider Package, Version and Source lines
>      #
> @@ -51,11 +32,6 @@ debsrc_download() {
>      export rootfs_distro="$2"
>      mkdir -p "${DEBSRCDIR}"/"${rootfs_distro}"
>  
> -    debsrc_do_mounts "${rootfs}"
> -
> -    trap 'exit 1' INT HUP QUIT TERM ALRM USR1
> -    trap 'debsrc_undo_mounts "${rootfs}"' EXIT
> -
>      ( flock 9
>      set -e
>      printenv | grep -q BB_VERBOSE_LOGS && set -x
> @@ -96,13 +72,18 @@ debsrc_download() {
>          dscname="${src}_${version#*:}.dsc"
>          [ -f "${DEBSRCDIR}"/"${rootfs_distro}"/"${src}"/"${dscname}" ] || {
>              # use apt-get source to download sources in DEBSRCDIR
> -            sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
> -                sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"
> +            mkdir -p "${DEBSRCDIR}/${rootfs_distro}"/"${src}"
> +            rootfs_cmd \
> +                --bind "${DEBSRCDIR}" "/deb-src" \
> +                --bind "${rootfs}" "${rootfs}" \
> +                --chdir "/deb-src/${rootfs_distro}/${src}" \
> +                -- \
> +                apt-get -o APT::Architecture=${DISTRO_ARCH} \
> +                        -o Dir="${rootfs}" -y --download-only \
> +                        --only-source source "${src}=${version}"
>          }
>      done
>      ) 9>"${DEBSRCDIR}/${rootfs_distro}.lock"

This lock was previously owned by root, hence it must be manually
cleaned when updating to this series. Otherwise the following error
happens:

/work/build/tmp/work/debian-bookworm-arm64/isar-mmdebstrap-host/1.0-
r0/temp/run.do_bootstrap.1528: 262: cannot create
/work/build/downloads/deb/debian-bookworm.lock: Permission denied

Apart from that, I'm wondering why we don't delete this lock in one of
the cleanup handlers. We anyways can only run a single bitbake instance
per build dir at a time.

Felix

> -
> -    debsrc_undo_mounts "${rootfs}"
>  }
>  
>  dbg_pkgs_download() {
> -- 
> 2.47.3
> 
> -- 
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250925065433.4180883-3-cedric.hombourger%40siemens.com.

-- 
Siemens AG
Linux Expert Center
Friedrich-Ludwig-Bauer-Str. 3
85748 Garching, Germany

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/87f6ac8b73d31298591e61572426cc5a17218942.camel%40siemens.com.

Re: [PATCH v4 0/4] non-privileged commands in chroot

['MOESSBAUER, Felix' via isar-users]

On Thu, 2025-09-25 at 08:54 +0200, 'Cedric Hombourger' via isar-users
wrote:
> When building root filesystems for foreign architectures with package source
> caching enabled, apt operations are executed within the rootfs through QEMU
> emulation. This results in significantly degraded performance, particularly
> when downloading source packages sequentially.
> 
> This patch series introduces a new wrapper function that enables native
> command execution against a rootfs while preserving special mount points
> (such as /isar-apt). The approach:
> 
> - Improves build performance for foreign architecture builds
> - Maintains filesystem isolation using bubblewrap
> - Preserves access to special mount points required by isar
> 
> Additional notes:
> - rootfs_cmd may be used to run commands from the host root file-system:
>   use with extreme care to avoid host contamination problems.
> - mmdebstrap already calls apt-get of the host to download packages (in
>   other words, a build of a bookworm image from a trixie host will
>   result in mmdebstrap (from trixie), call apt-get (from trixie) to
>   download bookworm packages. This is the behavior we have before and
>   after these changes.
> - With these changes and when caching of Debian source packages is
>   enabled/requested, Isar will use apt-get of the host to download
>   source packages (it will however do this from a bubblewrap'ed
>   environment to avoid a non-required privilege elevation; Isar has
>   many but we need to start from somewhere).
> 
> Testing:
> - Tested against 9e62337953fbb8371c846c44e8a99d62a8d220ba
> - Basic smoke tests performed successfully (citest.py -t fast)
> - Performance improvements observed in source package acquisition
> - Tested with various foreign architecture configurations

Thanks for bringing this up again. I successfully tested this series on
(amd64, arm64) x (bookworm, trixie).

Tested-by: Felix Moessbauer <felix.moessbauer@siemens.com>

Felix

-- 
Siemens AG
Linux Expert Center
Friedrich-Ludwig-Bauer-Str. 3
85748 Garching, Germany

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/7726069416bb27b02244de47750ffeceb64bd6ca.camel%40siemens.com.

Re: [PATCH v4 1/4] rootfs: introduce wrapper to run commands against a rootfs

[Andreas Naumann]

Hi Cedric,

Am 25.09.25 um 08:54 schrieb 'Cedric Hombourger' via isar-users:
> From: "cedric.hombourger@siemens.com" <cedric.hombourger@siemens.com>
>
> "sudo chroot" is used in several places to run commands inside rootfs
> directories constructed by Isar. There are cases where a command could
> be used without elevated privileges as long as special folders such as
> /isar-apt are mounted (they are often referenced as /isar-apt in
> configuration files found in the target rootfs). For such cases,
> bubblewrap may be used to create a non-privileged namespace (either
> in a bare/native environment or within a docker/podman container)
> where the command will be executed as if chroot had been used. The
> rootfs may also be the host root file-system: this should however
> be used with care to avoid host contamination problems (note: Isar
> already relies on a number of host tools).

Thank you for the respin! This opens interesting possibilities.

I'm currently trying run the build in a container with less than full 
privileges and it seems that using --cap-add=SYS_ADMIN is more or less 
is sufficient.

Now with your patch applied, I'm unfortunately greeted with

   | bwrap: pivot_root: Operation not permitted

This is caused by the docker (podman on Ubuntu 24.04 doesnt show the 
issue) default seccomp profile which denies the pivot_root syscall, 
among others. Of course it could be overcome by adding --security-opt 
seccomp=unconfined. Or better a more nuanced seccomp profile.

However, the ultimate goal of reducing the needed privileges would be to 
run docker out of the box without the need to extend capabilites or 
adding exceptions in the default MAC profiles. So, in this regard bwrap, 
which I had thought to be a key to overcome the bind-mount restriction 
when switching to rootless containers, adds something to take care of.

I dont know if this is something that's even on your roadmap, and for 
sure it's nothing that should stop this series, I just wanted to share 
the information.

Actually, I just found there's an open issue with even a workaround to 
avoid pivot_root: https://github.com/containers/bubblewrap/issues/592, 
so maybe it can be solved easily later.

regards,
Andreas



>
> Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
> ---
>   RECIPE-API-CHANGELOG.md     |  8 +++++
>   doc/user_manual.md          |  1 +
>   meta/classes/rootfs.bbclass | 67 +++++++++++++++++++++++++++++++++++++
>   3 files changed, 76 insertions(+)
>
> diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
> index 92e7811c..53e650d4 100644
> --- a/RECIPE-API-CHANGELOG.md
> +++ b/RECIPE-API-CHANGELOG.md
> @@ -741,3 +741,11 @@ By setting `MS_TPM_20_REF_DIR` in an optee-ftpm recipe, it is now possible to
>   use the new optee_ftpm code base from the OP-TEE project. That variable has to
>   point to a subdir in `WORKDIR` which contains the unpacked ms-tpm-20-ref source
>   code.
> +
> +### Require bubblewrap to run non-privileged commands with bind-mounts
> +
> +Isar occasionally needs to run commands within root file-systems that it
> +builds and with several bind-mounts (e.g. /isar-apt). bubblewrap may be
> +used in Isar classes instead of `sudo chroot` to avoid unecessary privilege
> +elevations (when we "just" need to chroot but do not require root). It is
> +pre-installed in kas-container version 4.8 (or later).
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 67f91973..be89ce1d 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -75,6 +75,7 @@ Install the following packages:
>   ```
>   apt install \
>     binfmt-support \
> +  bubblewrap \
>     bzip2 \
>     mmdebstrap \
>     arch-test \
> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
> index ebe3bf4a..f740c6e1 100644
> --- a/meta/classes/rootfs.bbclass
> +++ b/meta/classes/rootfs.bbclass
> @@ -34,6 +34,73 @@ export LANG = "C"
>   export LANGUAGE = "C"
>   export LC_ALL = "C"
>   
> +# Execute a command against a rootfs and with isar-apt bind-mounted.
> +# Additional mounts may be specified using --bind <source> <target> and a
> +# custom directory for the command to be executed with --chdir <dir>. The
> +# command is assumed to follow the special "--" argument. This would replace
> +# "sudo chroot" calls especially when a native command may be used instead of
> +# chroot'ed command and without elevated privileges (the command will likely
> +# take the rootfs as argument; e.g. apt-get -o Dir=${ROOTFSDIR}). If the
> +# optional rootfs argument is omitted, the host rootfs will be used (e.g. to
> +# run native commands): this should be used with care.
> +#
> +# Usage: rootfs_cmd [options] [rootfs] -- command
> +#
> +rootfs_cmd() {
> +    set -- "$@"
> +    bwrap_args="--bind ${REPO_ISAR_DIR}/${DISTRO} /isar-apt"
> +    bwrap_binds=""
> +    bwrap_rootfs=""
> +
> +    while [ "${#}" -gt "0" ] && [ "${1}" != "--" ]; do
> +        case "${1}" in
> +            --bind)
> +                if [ "${#}" -lt "3" ]; then
> +                    bbfatal "--bind requires two arguments"
> +                fi
> +                bwrap_binds="${bwrap_binds} --bind ${2} ${3}"
> +                shift 3
> +                ;;
> +            --chdir)
> +                if [ "${#}" -lt "2" ]; then
> +                    bbfatal "${1} requires an argument"
> +                fi
> +                bwrap_args="${bwrap_args} ${1} ${2}"
> +                shift 2
> +                ;;
> +            -*)
> +                bbfatal "${1} is not a supported option!"
> +                ;;
> +            *)
> +                if [ -z "${bwrap_rootfs}" ]; then
> +                    bwrap_rootfs="${1}"
> +                    shift
> +                else
> +                    bbfatal "unexpected argument '${1}'"
> +                fi
> +                ;;
> +        esac
> +    done
> +
> +    if [ -n "${bwrap_rootfs}" ]; then
> +        bwrap_args="${bwrap_args} --bind ${bwrap_rootfs} /"
> +    fi
> +
> +    if [ "${#}" -le "1" ] || [ "${1}" != "--" ]; then
> +        bbfatal "no command specified (missing --)"
> +    fi
> +    shift  # remove "--", command and its arguments follows
> +
> +    for ro_d in bin etc lib lib64 sys usr var; do
> +        [ -d ${bwrap_rootfs}/${ro_d} ] || continue
> +        bwrap_args="${bwrap_args} --ro-bind ${bwrap_rootfs}/${ro_d} /${ro_d}"
> +    done
> +
> +    bwrap --unshare-user --unshare-pid ${bwrap_args} \
> +        --dev-bind /dev /dev --proc /proc --tmpfs /tmp \
> +        ${bwrap_binds} -- "${@}"
> +}
> +
>   rootfs_do_mounts[weight] = "3"
>   rootfs_do_mounts() {
>       sudo -s <<'EOSUDO'

-- 
Andreas Naumann

emlix GmbH
Headquarters: Berliner Str. 12, 37073 Goettingen, Germany
Phone +49 (0)551 30664-0, e-mail info@emlix.com
District Court of Goettingen, Registry Number HR B 3160
Managing Directors: Heike Jordan, Dr. Uwe Kracke
VAT ID No. DE 205 198 055
Office Berlin: Panoramastr. 1, 10178 Berlin, Germany
Office Bonn: Bachstr. 6, 53115 Bonn, Germany
http://www.emlix.com

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/6e6ee857-58a9-4570-aa7d-7ba071c725f3%40emlix.com.

Re: [PATCH v4 3/4] bootstrap: create lock for downloads/deb without sudo

[Andreas Naumann]

Am 25.09.25 um 08:54 schrieb 'Cedric Hombourger' via isar-users:
> From: "cedric.hombourger@siemens.com" <cedric.hombourger@siemens.com>
>
> The syncin/syncout commands passed to mmdebstrap will create a lock
> file in downloads/deb if it does not exist. As mmdebstrap is being
> executed as root, the lock would also be owned by root and this will
> cause problems for rootless commands that may be executed later (such
> as downloading of Debian packages). Create the lock file without
> sudo prior to running mmdebstrap for it to be owned by the build user
> rather than root.
>
> Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
> ---
>   meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 4 ++++
>   1 file changed, 4 insertions(+)
>
> diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
> index 931f6f13..b2de61ad 100644
> --- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
> +++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
> @@ -181,6 +181,10 @@ do_bootstrap() {
>                                    && sudo umount $base_apt_tmp \
>                                    && rm -rf --one-file-system $base_apt_tmp' EXIT
>   
> +    # Create lock file so that it is owned by the user running the build (not root)
> +    mkdir -p ${DEBDIR}
> +    touch ${DEB_DL_LOCK}
> +

Tested-by: Andreas Naumann <anaumann@emlix.com>

Actually by a collegue of mine.

>       sudo TMPDIR="${BOOTSTRAP_TMPDIR}" mmdebstrap $bootstrap_args \
>                      $arch_param \
>                      --mode=unshare \

-- 
Andreas Naumann

emlix GmbH
Headquarters: Berliner Str. 12, 37073 Goettingen, Germany
Phone +49 (0)551 30664-0, e-mail info@emlix.com
District Court of Goettingen, Registry Number HR B 3160
Managing Directors: Heike Jordan, Dr. Uwe Kracke
VAT ID No. DE 205 198 055
Office Berlin: Panoramastr. 1, 10178 Berlin, Germany
Office Bonn: Bachstr. 6, 53115 Bonn, Germany
http://www.emlix.com

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/3a8541e0-c79d-4b50-bbf4-ab5c1198a94f%40emlix.com.