public inbox for isar-users@googlegroups.com
 help / color / mirror / Atom feed
* [PATCH v3 0/6] non-privileged commands in chroot
@ 2025-06-25 19:37 'Cedric Hombourger' via isar-users
  2025-06-25 19:37 ` [PATCH v3 1/6] rootfs: introduce wrapper to run commands against a rootfs 'Cedric Hombourger' via isar-users
                   ` (5 more replies)
  0 siblings, 6 replies; 7+ messages in thread
From: 'Cedric Hombourger' via isar-users @ 2025-06-25 19:37 UTC (permalink / raw)
  To: isar-users; +Cc: felix.moessbauer, Cedric Hombourger

When building root filesystems for foreign architectures with package source
caching enabled, apt operations are executed within the rootfs through QEMU
emulation. This results in significantly degraded performance, particularly
when downloading source packages sequentially.

This patch series introduces a new wrapper function that enables native
command execution against a rootfs while preserving special mount points
(such as /isar-apt). The approach:

- Improves build performance for foreign architecture builds
- Maintains filesystem isolation using bubblewrap
- Preserves access to special mount points required by isar

Testing:
- Basic smoke tests performed successfully (citest.py -t fast)
- Performance improvements observed in source package acquisition
- Tested with various foreign architecture configurations

Dependencies:
- Adds bubblewrap as a new host tool requirement
- Uses kas-container 4.8.0 or later (see [1])

Changes since v2 patch:
  - rootfs_install_pkgs_download will no longer use sudo to run
    apt-get install --download-only. This was added to further
    demonstrate/test rootfs_cmd in existing Isar code.

Changes since v1 patch:
  - Rebase (resolve RECIPE-API-CHANGELOG.md merge conflicts)
  - Prefix rootfs variable in rootfs_cmd with bwrap to avoid clashes

Changes since RFC patch:
  - Let caller decide where to bind-mount the rootfs to
  - Make the rootfs argument optional
  - Support 32-bit rootfs (no lib64 there)

(Re-)validated with "citest.py -t fast" (using kas-container 4.8.1):
 
 JOB ID     : 2724be97c6711e046fbc2169823c293dc99cd97c
 JOB LOG    : avocado/job-results/job-2025-06-25T15.51-2724be9/job.log
  (01/19) citest.py:DevTest.test_dev: STARTED
  (01/19) citest.py:DevTest.test_dev:  PASS (1573.34 s)
  (02/19) citest.py:DevTest.test_dev_apps: STARTED
  (02/19) citest.py:DevTest.test_dev_apps:  PASS (2158.85 s)
  (03/19) citest.py:DevTest.test_dev_rebuild: STARTED
  (03/19) citest.py:DevTest.test_dev_rebuild:  PASS (349.73 s)
  (04/19) citest.py:DevTest.test_dev_run_amd64_bookworm: STARTED
  (04/19) citest.py:DevTest.test_dev_run_amd64_bookworm:  PASS (77.79 s)
  (05/19) citest.py:DevTest.test_dev_run_arm64_bookworm: STARTED
  (05/19) citest.py:DevTest.test_dev_run_arm64_bookworm:  PASS (55.12 s)
  (06/19) citest.py:DevTest.test_dev_run_arm_bookworm: STARTED
  (06/19) citest.py:DevTest.test_dev_run_arm_bookworm:  PASS (58.94 s)
  (07/19) citest.py:CrossTest.test_cross: STARTED
  (07/19) citest.py:CrossTest.test_cross:  PASS (1912.25 s)
  (08/19) citest.py:CrossTest.test_cross_debsrc: STARTED
  (08/19) citest.py:CrossTest.test_cross_debsrc:  PASS (2933.62 s)
  (09/19) citest.py:CrossTest.test_cross_kselftest: STARTED
  (09/19) citest.py:CrossTest.test_cross_kselftest:  PASS (2024.26 s)
  (10/19) citest.py:CrossTest.test_cross_rpi: STARTED
  (10/19) citest.py:CrossTest.test_cross_rpi:  PASS (1543.77 s)
  (11/19) citest.py:VmBootTestFast.test_arm_bullseye: STARTED
  (11/19) citest.py:VmBootTestFast.test_arm_bullseye:  PASS (64.33 s)
  (12/19) citest.py:VmBootTestFast.test_arm_bullseye_example_module: STARTED
  (12/19) citest.py:VmBootTestFast.test_arm_bullseye_example_module:  PASS (12.72 s)
  (13/19) citest.py:VmBootTestFast.test_arm_bullseye_getty_target: STARTED
  (13/19) citest.py:VmBootTestFast.test_arm_bullseye_getty_target:  PASS (10.18 s)
  (14/19) citest.py:VmBootTestFast.test_arm_buster: STARTED
  (14/19) citest.py:VmBootTestFast.test_arm_buster:  PASS (57.01 s)
  (15/19) citest.py:VmBootTestFast.test_arm_buster_getty_target: STARTED
  (15/19) citest.py:VmBootTestFast.test_arm_buster_getty_target:  PASS (9.73 s)
  (16/19) citest.py:VmBootTestFast.test_arm_buster_example_module: STARTED
  (16/19) citest.py:VmBootTestFast.test_arm_buster_example_module:  PASS (10.39 s)
  (17/19) citest.py:VmBootTestFast.test_arm_bookworm: STARTED
  (17/19) citest.py:VmBootTestFast.test_arm_bookworm:  PASS (82.93 s)
  (18/19) citest.py:VmBootTestFast.test_arm_bookworm_example_module: STARTED
  (18/19) citest.py:VmBootTestFast.test_arm_bookworm_example_module:  PASS (30.40 s)
  (19/19) citest.py:VmBootTestFast.test_arm_bookworm_getty_target: STARTED
  (19/19) citest.py:VmBootTestFast.test_arm_bookworm_getty_target:  PASS (11.59 s)
 RESULTS    : PASS 19 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
 JOB TIME   : 13003.86 s
 
Cedric Hombourger (5):
  rootfs: introduce wrapper to run commands against a rootfs
  deb-dl-dir: optimize caching of source packages using apt natively
  image-postproc-extension: refactor systemd version checks
  image-postproc-extension: extract systemd's version using rootfs_cmd
  bootstrap: create lock for downloads/deb without sudo
  rootfs: do not get elevated privileges when downloading packages

 RECIPE-API-CHANGELOG.md                       |  7 ++
 doc/user_manual.md                            |  1 +
 meta/classes/deb-dl-dir.bbclass               | 58 ++++++-------
 meta/classes/image-postproc-extension.bbclass | 12 +--
 meta/classes/rootfs.bbclass                   | 83 ++++++++++++++++++-
 .../isar-mmdebstrap/isar-mmdebstrap.inc       |  4 +
 6 files changed, 126 insertions(+), 39 deletions(-)

-- 
2.39.5

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250625193748.2681-1-cedric.hombourger%40siemens.com.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* [PATCH v3 1/6] rootfs: introduce wrapper to run commands against a rootfs
  2025-06-25 19:37 [PATCH v3 0/6] non-privileged commands in chroot 'Cedric Hombourger' via isar-users
@ 2025-06-25 19:37 ` 'Cedric Hombourger' via isar-users
  2025-06-25 19:37 ` [PATCH v3 2/6] deb-dl-dir: optimize caching of source packages using apt natively 'Cedric Hombourger' via isar-users
                   ` (4 subsequent siblings)
  5 siblings, 0 replies; 7+ messages in thread
From: 'Cedric Hombourger' via isar-users @ 2025-06-25 19:37 UTC (permalink / raw)
  To: isar-users; +Cc: felix.moessbauer, Cedric Hombourger

"sudo chroot" is used in several places to run commands inside rootfs
directories constructed by Isar. There are cases where a command could
be used without elevated privileges as long as special folders such as
/isar-apt are mounted (they are often referenced as /isar-apt in
configuration files found in the target rootfs). For such cases,
bubblewrap may be used to create a non-privileged namespace (either
in a bare/native environment or within a docker/podman container)
where the command will be executed as if chroot had been used. The
rootfs may also be the host root file-system: this should however
be used with care to avoid host contamination problems (note: Isar
already relies on a number of host tools).

Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
---
 RECIPE-API-CHANGELOG.md     |  7 ++++
 doc/user_manual.md          |  1 +
 meta/classes/rootfs.bbclass | 67 +++++++++++++++++++++++++++++++++++++
 3 files changed, 75 insertions(+)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 8468717d..18b90555 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -727,3 +727,10 @@ Changes in next
 
 This was never documented and never had practical relevance. `oci-archive` is
 the useful OCI image format that can be imported, e.g., by podman.
+
+### Require bubblewrap to run non-privileged commands with bind-mounts
+
+Isar occasionally needs to run commands within root file-systems that it
+builds and with several bind-mounts (e.g. /isar-apt). bubblewrap may be
+used in Isar classes instead of `sudo chroot`. It is pre-installed in
+kas-container version 4.8 (or later).
diff --git a/doc/user_manual.md b/doc/user_manual.md
index ca551a0d..a4fff34a 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -75,6 +75,7 @@ Install the following packages:
 ```
 apt install \
   binfmt-support \
+  bubblewrap \
   bzip2 \
   mmdebstrap \
   arch-test \
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 5f877962..429494ae 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -34,6 +34,73 @@ export LANG = "C"
 export LANGUAGE = "C"
 export LC_ALL = "C"
 
+# Execute a command against a rootfs and with isar-apt bind-mounted.
+# Additional mounts may be specified using --bind <source> <target> and a
+# custom directory for the command to be executed with --chdir <dir>. The
+# command is assumed to follow the special "--" argument. This would replace
+# "sudo chroot" calls especially when a native command may be used instead of
+# chroot'ed command and without elevated privileges (the command will likely
+# take the rootfs as argument; e.g. apt-get -o Dir=${ROOTFSDIR}). If the
+# optional rootfs argument is omitted, the host rootfs will be used (e.g. to
+# run native commands): this should be used with care.
+#
+# Usage: rootfs_cmd [options] [rootfs] -- command
+#
+rootfs_cmd() {
+    set -- "$@"
+    bwrap_args="--bind ${REPO_ISAR_DIR}/${DISTRO} /isar-apt"
+    bwrap_binds=""
+    bwrap_rootfs=""
+
+    while [ "${#}" -gt "0" ] && [ "${1}" != "--" ]; do
+        case "${1}" in
+            --bind)
+                if [ "${#}" -lt "3" ]; then
+                    bbfatal "--bind requires two arguments"
+                fi
+                bwrap_binds="${bwrap_binds} --bind ${2} ${3}"
+                shift 3
+                ;;
+            --chdir)
+                if [ "${#}" -lt "2" ]; then
+                    bbfatal "${1} requires an argument"
+                fi
+                bwrap_args="${bwrap_args} ${1} ${2}"
+                shift 2
+                ;;
+            -*)
+                bbfatal "${1} is not a supported option!"
+                ;;
+            *)
+                if [ -z "${bwrap_rootfs}" ]; then
+                    bwrap_rootfs="${1}"
+                    shift
+                else
+                    bbfatal "unexpected argument '${1}'"
+                fi
+                ;;
+        esac
+    done
+
+    if [ -n "${bwrap_rootfs}" ]; then
+        bwrap_args="${bwrap_args} --bind ${bwrap_rootfs} /"
+    fi
+
+    if [ "${#}" -le "1" ] || [ "${1}" != "--" ]; then
+        bbfatal "no command specified (missing --)"
+    fi
+    shift  # remove "--", command and its arguments follows
+
+    for ro_d in bin etc lib lib64 sys usr var; do
+        [ -d ${bwrap_rootfs}/${ro_d} ] || continue
+        bwrap_args="${bwrap_args} --ro-bind ${bwrap_rootfs}/${ro_d} /${ro_d}"
+    done
+
+    bwrap --unshare-user --unshare-pid ${bwrap_args} \
+        --dev-bind /dev /dev --proc /proc --tmpfs /tmp \
+        ${bwrap_binds} -- "${@}"
+}
+
 rootfs_do_mounts[weight] = "3"
 rootfs_do_mounts() {
     sudo -s <<'EOSUDO'
-- 
2.39.5

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250625193748.2681-2-cedric.hombourger%40siemens.com.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* [PATCH v3 2/6] deb-dl-dir: optimize caching of source packages using apt natively
  2025-06-25 19:37 [PATCH v3 0/6] non-privileged commands in chroot 'Cedric Hombourger' via isar-users
  2025-06-25 19:37 ` [PATCH v3 1/6] rootfs: introduce wrapper to run commands against a rootfs 'Cedric Hombourger' via isar-users
@ 2025-06-25 19:37 ` 'Cedric Hombourger' via isar-users
  2025-06-25 19:37 ` [PATCH v3 3/6] image-postproc-extension: refactor systemd version checks 'Cedric Hombourger' via isar-users
                   ` (3 subsequent siblings)
  5 siblings, 0 replies; 7+ messages in thread
From: 'Cedric Hombourger' via isar-users @ 2025-06-25 19:37 UTC (permalink / raw)
  To: isar-users; +Cc: felix.moessbauer, Cedric Hombourger

source package are downloaded by entering the target rootfs and run
apt there. For foreign architectures, this results in apt being
executed under QEMU and leads to poor performance. By using the
recently introduced rootfs_native_cmd command wrapper, apt will be
executed natively against the target rootfs and without elevated
privileges. For our test work-load, caching was reduced from more
than 10 hours to an hour. Performance is also more consistent as
it will no longer depend as to when bitbake kicks caching of
source packages for foreign architecture rootfs vs rootfs for the
host (in multiconfig builds).

Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
---
 meta/classes/deb-dl-dir.bbclass | 37 ++++++++-------------------------
 1 file changed, 9 insertions(+), 28 deletions(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 3f560da4..7026f4f4 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -5,25 +5,6 @@
 
 inherit repository
 
-debsrc_do_mounts() {
-    sudo -s <<EOSUDO
-    set -e
-    mkdir -p "${1}/deb-src"
-    mountpoint -q "${1}/deb-src" || \
-    mount -o bind,private "${DEBSRCDIR}" "${1}/deb-src"
-EOSUDO
-}
-
-debsrc_undo_mounts() {
-    sudo -s <<EOSUDO
-    set -e
-    mkdir -p "${1}/deb-src"
-    mountpoint -q "${1}/deb-src" && \
-    umount "${1}/deb-src"
-    rm -rf "${1}/deb-src"
-EOSUDO
-}
-
 debsrc_source_version_filter() {
     # Filter the input to only consider Package, Version and Source lines
     #
@@ -51,11 +32,6 @@ debsrc_download() {
     export rootfs_distro="$2"
     mkdir -p "${DEBSRCDIR}"/"${rootfs_distro}"
 
-    debsrc_do_mounts "${rootfs}"
-
-    trap 'exit 1' INT HUP QUIT TERM ALRM USR1
-    trap 'debsrc_undo_mounts "${rootfs}"' EXIT
-
     ( flock 9
     set -e
     printenv | grep -q BB_VERBOSE_LOGS && set -x
@@ -89,13 +65,18 @@ debsrc_download() {
         dscname="${src}_${version#*:}.dsc"
         [ -f "${DEBSRCDIR}"/"${rootfs_distro}"/"${src}"/"${dscname}" ] || {
             # use apt-get source to download sources in DEBSRCDIR
-            sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
-                sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"
+            mkdir -p "${DEBSRCDIR}/${rootfs_distro}"/"${src}"
+            rootfs_cmd \
+                --bind "${DEBSRCDIR}" "/deb-src" \
+                --bind "${rootfs}" "${rootfs}" \
+                --chdir "/deb-src/${rootfs_distro}/${src}" \
+                -- \
+                apt-get -o APT::Architecture=${DISTRO_ARCH} \
+                        -o Dir="${rootfs}" -y --download-only \
+                        --only-source source "${src}=${version}"
         }
     done
     ) 9>"${DEBSRCDIR}/${rootfs_distro}.lock"
-
-    debsrc_undo_mounts "${rootfs}"
 }
 
 dbg_pkgs_download() {
-- 
2.39.5

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250625193748.2681-3-cedric.hombourger%40siemens.com.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* [PATCH v3 3/6] image-postproc-extension: refactor systemd version checks
  2025-06-25 19:37 [PATCH v3 0/6] non-privileged commands in chroot 'Cedric Hombourger' via isar-users
  2025-06-25 19:37 ` [PATCH v3 1/6] rootfs: introduce wrapper to run commands against a rootfs 'Cedric Hombourger' via isar-users
  2025-06-25 19:37 ` [PATCH v3 2/6] deb-dl-dir: optimize caching of source packages using apt natively 'Cedric Hombourger' via isar-users
@ 2025-06-25 19:37 ` 'Cedric Hombourger' via isar-users
  2025-06-25 19:37 ` [PATCH v3 4/6] image-postproc-extension: extract systemd's version using rootfs_cmd 'Cedric Hombourger' via isar-users
                   ` (2 subsequent siblings)
  5 siblings, 0 replies; 7+ messages in thread
From: 'Cedric Hombourger' via isar-users @ 2025-06-25 19:37 UTC (permalink / raw)
  To: isar-users; +Cc: felix.moessbauer, Cedric Hombourger

Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
---
 meta/classes/image-postproc-extension.bbclass | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/meta/classes/image-postproc-extension.bbclass b/meta/classes/image-postproc-extension.bbclass
index 991bac4c..0af588d8 100644
--- a/meta/classes/image-postproc-extension.bbclass
+++ b/meta/classes/image-postproc-extension.bbclass
@@ -53,12 +53,17 @@ image_postprocess_mark() {
         --build-id "${BUILD_ID}" --variant "${DESCRIPTION}" --version "${PV}"
 }
 
+# Use dpkg to find out which version of systemd is installed into the image or reports "0"
+image_systemd_version() {
+    sudo chroot ${IMAGE_ROOTFS} dpkg-query --showformat='${source:Upstream-Version}' --show systemd || echo "0"
+}
+
 ROOTFS_POSTPROCESS_COMMAND =+ "image_postprocess_machine_id"
 image_postprocess_machine_id() {
     # systemd(1) takes care of recreating the machine-id on first boot
     # for systemd < v247, set to empty string, else set to uninitialized
     # (required if initramfs with ro root is used)
-    SYSTEMD_VERSION=$( sudo chroot ${IMAGE_ROOTFS} dpkg-query --showformat='${source:Upstream-Version}' --show systemd || echo "0" )
+    SYSTEMD_VERSION=$( image_systemd_version )
     MACHINE_ID="uninitialized"
     if dpkg --compare-versions "$SYSTEMD_VERSION" "lt" "247"; then
         MACHINE_ID=""
@@ -82,10 +87,7 @@ image_postprocess_sshd_key_regen() {
 
 ROOTFS_POSTPROCESS_COMMAND =+ "image_posprocess_disable_systemd_firstboot"
 image_posprocess_disable_systemd_firstboot() {
-    SYSTEMD_VERSION=$(sudo chroot '${ROOTFSDIR}' dpkg-query \
-        --showformat='${source:Upstream-Version}' \
-        --show systemd || echo "0" )
-
+    SYSTEMD_VERSION=$( image_systemd_version )
     if dpkg --compare-versions "$SYSTEMD_VERSION" "ge" "251"; then
         sudo chroot '${ROOTFSDIR}' systemctl mask systemd-firstboot
         if ! cmd_output=$(sudo chroot '${ROOTFSDIR}' systemd-firstboot \
-- 
2.39.5

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250625193748.2681-4-cedric.hombourger%40siemens.com.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* [PATCH v3 4/6] image-postproc-extension: extract systemd's version using rootfs_cmd
  2025-06-25 19:37 [PATCH v3 0/6] non-privileged commands in chroot 'Cedric Hombourger' via isar-users
                   ` (2 preceding siblings ...)
  2025-06-25 19:37 ` [PATCH v3 3/6] image-postproc-extension: refactor systemd version checks 'Cedric Hombourger' via isar-users
@ 2025-06-25 19:37 ` 'Cedric Hombourger' via isar-users
  2025-06-25 19:37 ` [PATCH v3 5/6] bootstrap: create lock for downloads/deb without sudo 'Cedric Hombourger' via isar-users
  2025-06-25 19:37 ` [PATCH v3 6/6] rootfs: do not get elevated privileges when downloading packages 'Cedric Hombourger' via isar-users
  5 siblings, 0 replies; 7+ messages in thread
From: 'Cedric Hombourger' via isar-users @ 2025-06-25 19:37 UTC (permalink / raw)
  To: isar-users; +Cc: felix.moessbauer, Cedric Hombourger

From: Cedric Hombourger' via isar-users <isar-users@googlegroups.com>

Elevated privileges are not required to query the rootfs for the version
of systemd: replace "sudo chroot" with "rootfs_cmd"

Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
---
 meta/classes/image-postproc-extension.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/image-postproc-extension.bbclass b/meta/classes/image-postproc-extension.bbclass
index 0af588d8..21dcfccc 100644
--- a/meta/classes/image-postproc-extension.bbclass
+++ b/meta/classes/image-postproc-extension.bbclass
@@ -55,7 +55,7 @@ image_postprocess_mark() {
 
 # Use dpkg to find out which version of systemd is installed into the image or reports "0"
 image_systemd_version() {
-    sudo chroot ${IMAGE_ROOTFS} dpkg-query --showformat='${source:Upstream-Version}' --show systemd || echo "0"
+    rootfs_cmd ${IMAGE_ROOTFS} -- dpkg-query --showformat='${source:Upstream-Version}' --show systemd || echo "0"
 }
 
 ROOTFS_POSTPROCESS_COMMAND =+ "image_postprocess_machine_id"
-- 
2.39.5

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250625193748.2681-5-cedric.hombourger%40siemens.com.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* [PATCH v3 5/6] bootstrap: create lock for downloads/deb without sudo
  2025-06-25 19:37 [PATCH v3 0/6] non-privileged commands in chroot 'Cedric Hombourger' via isar-users
                   ` (3 preceding siblings ...)
  2025-06-25 19:37 ` [PATCH v3 4/6] image-postproc-extension: extract systemd's version using rootfs_cmd 'Cedric Hombourger' via isar-users
@ 2025-06-25 19:37 ` 'Cedric Hombourger' via isar-users
  2025-06-25 19:37 ` [PATCH v3 6/6] rootfs: do not get elevated privileges when downloading packages 'Cedric Hombourger' via isar-users
  5 siblings, 0 replies; 7+ messages in thread
From: 'Cedric Hombourger' via isar-users @ 2025-06-25 19:37 UTC (permalink / raw)
  To: isar-users; +Cc: felix.moessbauer, Cedric Hombourger

The syncin/syncout commands passed to mmdebstrap will create a lock
file in downloads/deb if it does not exist. As mmdebstrap is being
executed as root, the lock would also be owned by root and this will
cause problems for rootless commands that may be executed later (such
as downloading of Debian packages). Create the lock file without
sudo prior to running mmdebstrap for it to be owned by the build user
rather than root.

Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
---
 meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
index 931f6f13..b2de61ad 100644
--- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
+++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
@@ -181,6 +181,10 @@ do_bootstrap() {
                                  && sudo umount $base_apt_tmp \
                                  && rm -rf --one-file-system $base_apt_tmp' EXIT
 
+    # Create lock file so that it is owned by the user running the build (not root)
+    mkdir -p ${DEBDIR}
+    touch ${DEB_DL_LOCK}
+
     sudo TMPDIR="${BOOTSTRAP_TMPDIR}" mmdebstrap $bootstrap_args \
                    $arch_param \
                    --mode=unshare \
-- 
2.39.5

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250625193748.2681-6-cedric.hombourger%40siemens.com.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* [PATCH v3 6/6] rootfs: do not get elevated privileges when downloading packages
  2025-06-25 19:37 [PATCH v3 0/6] non-privileged commands in chroot 'Cedric Hombourger' via isar-users
                   ` (4 preceding siblings ...)
  2025-06-25 19:37 ` [PATCH v3 5/6] bootstrap: create lock for downloads/deb without sudo 'Cedric Hombourger' via isar-users
@ 2025-06-25 19:37 ` 'Cedric Hombourger' via isar-users
  5 siblings, 0 replies; 7+ messages in thread
From: 'Cedric Hombourger' via isar-users @ 2025-06-25 19:37 UTC (permalink / raw)
  To: isar-users; +Cc: felix.moessbauer, Cedric Hombourger

Use rootfs_cmd() to run "apt-get install --download-only" without sudo.
This requires /var/cache/apt/archives/ to be writable by the build
user: change ownership while populating that folder with previously
downloaded packages (those in downloads/deb/).

Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
---
 meta/classes/deb-dl-dir.bbclass | 21 ++++++++++++++++++---
 meta/classes/rootfs.bbclass     | 16 +++++++++++++---
 2 files changed, 31 insertions(+), 6 deletions(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 7026f4f4..7fe052ef 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -100,9 +100,24 @@ dbg_pkgs_download() {
 deb_dl_dir_import() {
     export pc="${DEBDIR}/${2}"
     export rootfs="${1}"
-    sudo mkdir -p "${rootfs}"/var/cache/apt/archives/
+    export uid=$(id -u)
+    export gid=$(id -g)
+
+    # let our unprivileged user place downloaded packages in /var/cache/apt/archives/
+    sudo -Es << '    EOSUDO'
+        mkdir -p "${rootfs}"/var/cache/apt/archives/partial/
+        touch "${rootfs}"/var/cache/apt/archives/lock
+        chown -R ${uid}:${gid} "${rootfs}"/var/cache/apt/archives/
+    EOSUDO
+
+    # nothing to copy if download directory does not exist just yet
     [ ! -d "${pc}" ] && return 0
-    flock -s "${pc}".lock sudo -Es << 'EOSUDO'
+
+    # attempt to create hard-links for .deb files from downloads/ into
+    # /var/cache/apt/archives/ so apt will only download packages we
+    # have not yet downloaded. perform a regular copy whenever hard-links
+    # cannot be created
+    ( flock 9
         set -e
         printenv | grep -q BB_VERBOSE_LOGS && set -x
 
@@ -111,7 +126,7 @@ deb_dl_dir_import() {
             ln -Pf -t "${rootfs}"/var/cache/apt/archives/ "$p" 2>/dev/null ||
                 cp -n --no-preserve=owner -t "${rootfs}"/var/cache/apt/archives/ "$p"
         done
-EOSUDO
+    ) 9>"${pc}".lock
 }
 
 deb_dl_dir_export() {
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 429494ae..977bbec8 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -277,10 +277,20 @@ ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_download"
 rootfs_install_pkgs_download[weight] = "600"
 rootfs_install_pkgs_download[progress] = "custom:rootfs_progress.PkgsDownloadProgressHandler"
 rootfs_install_pkgs_download[isar-apt-lock] = "release-after"
-rootfs_install_pkgs_download[network] = "${TASK_USE_NETWORK_AND_SUDO}"
+rootfs_install_pkgs_download[network] = "${TASK_USE_NETWORK}"
 rootfs_install_pkgs_download() {
-    sudo -E chroot '${ROOTFSDIR}' \
-        /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only ${ROOTFS_PACKAGES}
+    mkdir -p "${WORKDIR}/dpkg"
+
+    # Use our own dpkg lock files rather than those in the rootfs since we are not root
+    # (this is safe as there are no concurrent apt/dpkg operations for that rootfs)
+    touch "${WORKDIR}/dpkg/lock" "${WORKDIR}/dpkg/lock-frontend"
+
+    # download packages using apt in a non-privileged namespace
+    rootfs_cmd --bind "${ROOTFSDIR}/var/cache/apt/archives" /var/cache/apt/archives \
+               --bind "${WORKDIR}/dpkg/lock" /var/lib/dpkg/lock \
+               --bind "${WORKDIR}/dpkg/lock-frontend" /var/lib/dpkg/lock-frontend \
+               ${ROOTFSDIR} \
+               -- /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only ${ROOTFS_PACKAGES}
 }
 
 ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT ??= ""
-- 
2.39.5

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250625193748.2681-7-cedric.hombourger%40siemens.com.

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2025-06-25 19:40 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2025-06-25 19:37 [PATCH v3 0/6] non-privileged commands in chroot 'Cedric Hombourger' via isar-users
2025-06-25 19:37 ` [PATCH v3 1/6] rootfs: introduce wrapper to run commands against a rootfs 'Cedric Hombourger' via isar-users
2025-06-25 19:37 ` [PATCH v3 2/6] deb-dl-dir: optimize caching of source packages using apt natively 'Cedric Hombourger' via isar-users
2025-06-25 19:37 ` [PATCH v3 3/6] image-postproc-extension: refactor systemd version checks 'Cedric Hombourger' via isar-users
2025-06-25 19:37 ` [PATCH v3 4/6] image-postproc-extension: extract systemd's version using rootfs_cmd 'Cedric Hombourger' via isar-users
2025-06-25 19:37 ` [PATCH v3 5/6] bootstrap: create lock for downloads/deb without sudo 'Cedric Hombourger' via isar-users
2025-06-25 19:37 ` [PATCH v3 6/6] rootfs: do not get elevated privileges when downloading packages 'Cedric Hombourger' via isar-users

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox