[PATCH v5 0/7] Migrate to mmdebstrap

[PATCH v5 0/7] Migrate to mmdebstrap

[Anton Mikanovich]

Switch Isar from debootstrap to mmdebstrap.

It makes isar-bootstrap target ~10..30% faster on both cross and native
rootfs types. Some measurements on qemuarm64-bookworm (required debs are
predownloaded and used during deb-dl-import):

- 20s vs 28s for "isar-bootstrap-host"
- 1m32s vs 1m48s for "isar-bootstrap-target

Also mmdebstrap support using SOURCE_DATE_EPOCH for images reproducibly
out of the box. But the main advantage is an ability to remove sudo for
debootstrapping later on (this needs "uidmap" package installed).

This change breaks Debian Buster support as host distro.

Changes since v4:
- Rebase on next.
- Migrate from apt-key to gpg.
- Fix Ubuntu and RPi targets.
- Cleanup debootstrap mentions.
- Fix commit messages.

Changes since v3:
- Rebase on latest next.
- Remove temporary patch 7 since kas:4.0 already includes mmdebstrap.

Changes since v2:
- Rebase on next.
- Fix commit messages.
- Update host requirements.

Changes since v1:
- Move mmdebstrap tmpdir to workdir.
- Fix kas building.
- Improve mounts cleanup.

Anton Mikanovich (6):
  meta: Preinstall apt for Ubuntu and RPi targets
  isar-bootstrap: Move preparations to hooks
  isar-bootstrap: Use tar output instead of directory
  user_manual.md: Replace debootstrap with mmdebstrap
  user_manual.md: Update minimal host requirements
  isar-bootstrap: Remove unused code

Uladzimir Bely (1):
  meta: Switch to mmdebstrap

 doc/user_manual.md                            |  41 +--
 meta-isar/conf/distro/ubuntu-common.inc       |   4 -
 meta/classes/rootfs.bbclass                   |  11 +-
 meta/conf/bitbake.conf                        |   2 +-
 .../isar-bootstrap/isar-bootstrap.inc         | 284 ++++++++----------
 5 files changed, 153 insertions(+), 189 deletions(-)

-- 
2.34.1

[PATCH v5 1/7] meta: Switch to mmdebstrap

[Anton Mikanovich]

From: Uladzimir Bely <ubely@ilbers.de>

Instead of debootstrap, use mmdebstrap alternative.

Internally, it uses apt and allows to debootstrap the distro from
multiple repositories.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta-isar/conf/distro/ubuntu-common.inc       |  4 --
 .../isar-bootstrap/isar-bootstrap.inc         | 64 ++++++++++---------
 2 files changed, 33 insertions(+), 35 deletions(-)

diff --git a/meta-isar/conf/distro/ubuntu-common.inc b/meta-isar/conf/distro/ubuntu-common.inc
index 6a990a89..afe96b73 100644
--- a/meta-isar/conf/distro/ubuntu-common.inc
+++ b/meta-isar/conf/distro/ubuntu-common.inc
@@ -18,10 +18,6 @@ HOST_DISTRO_BOOTSTRAP_KEYS += "${BOOTSTRAP_KEY}"
 
 # kernel package name is linux-image-generic in Ubuntu
 KERNEL_NAME = "generic"
-
-# that is what debootstrap_1.0.118ubuntu1 does anyways
-DISTRO_DEBOOTSTRAP_SCRIPT = "/usr/share/debootstrap/scripts/gutsy"
-
 DISTRO_CONFIG_SCRIPT ?= "debian-configscript.sh"
 
 DISTRO_KERNELS ?= " \
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 3477c2fb..9f1142ba 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -15,7 +15,6 @@ PV = "1.0"
 
 BOOTSTRAP_FOR_HOST ?= "0"
 
-DEBOOTSTRAP ?= "qemu-debootstrap"
 ROOTFSDIR = "${WORKDIR}/rootfs"
 APTPREFS = "${WORKDIR}/apt-preferences"
 APTSRCS = "${WORKDIR}/apt-sources"
@@ -23,12 +22,14 @@ APTSRCS_INIT = "${WORKDIR}/apt-sources-init"
 DISTRO_BOOTSTRAP_KEYFILES = ""
 THIRD_PARTY_APT_KEYFILES = ""
 DEPLOY_ISAR_BOOTSTRAP ?= ""
-DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"
+DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales,usrmerge"
 DISTRO_BOOTSTRAP_BASE_PACKAGES:append:gnupg = ",gnupg"
 DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = ",ca-certificates"
 DISTRO_VARS_PREFIX ?= "${@'HOST_' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}"
 BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO')}"
 BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}"
+BOOTSTRAP_DISTRO_ARCH = "${@d.getVar('HOST_ARCH' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO_ARCH')}"
+BOOTSTRAP_TMPDIR = "${WORKDIR}/tempdir"
 FILESEXTRAPATHS:append = ":${BBPATH}"
 
 inherit deb-dl-dir
@@ -39,7 +40,7 @@ python () {
     topdir = d.getVar("TOPDIR")
 
     # The cached repo key can be both for bootstrapping and apt package
-    # installation afterwards. However, debootstrap will include the key into
+    # installation afterwards. However, bootstrap will include the key into
     # the rootfs automatically thus the right place is distro_bootstrap_keys.
 
     if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
@@ -267,7 +268,7 @@ do_bootstrap[vardeps] += " \
     ISAR_ENABLE_COMPAT_ARCH \
     ${DISTRO_VARS_PREFIX}DISTRO_APT_SOURCES \
     "
-do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP}"
+do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP} ${BOOTSTRAP_TMPDIR}"
 do_bootstrap[depends] = "base-apt:do_cache isar-apt:do_cache_config"
 do_bootstrap[network] = "${TASK_USE_NETWORK_AND_SUDO}"
 
@@ -279,31 +280,35 @@ do_bootstrap() {
             bbfatal "${DISTRO_ARCH} does not have a compat arch"
         fi
     fi
-    debootstrap_args="--verbose --variant=minbase --include=${DISTRO_BOOTSTRAP_BASE_PACKAGES}"
+    bootstrap_args="--verbose --variant=minbase --include=${DISTRO_BOOTSTRAP_BASE_PACKAGES}"
     if [ -f "${DISTRO_BOOTSTRAP_KEYRING}" ]; then
-        debootstrap_args="$debootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}"
-    fi
-    if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" -a -z "${BASE_REPO_KEY}" ]; then
-        debootstrap_args="$debootstrap_args --no-check-gpg"
+        bootstrap_args="$bootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}"
     fi
     E="${@ isar_export_proxies(d)}"
-    export BOOTSTRAP_FOR_HOST debootstrap_args E
+    export BOOTSTRAP_FOR_HOST
 
-    sudo rm -rf --one-file-system "${ROOTFSDIR}"
     deb_dl_dir_import "${ROOTFSDIR}" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
+    sudo rm -rf --one-file-system "${ROOTFSDIR}"
+    mkdir -p "${ROOTFSDIR}"
+
+    if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then
+        export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
+    fi
+
+    arch_param="--arch=${BOOTSTRAP_DISTRO_ARCH},${DISTRO_ARCH}"
+
+    sudo TMPDIR="${BOOTSTRAP_TMPDIR}" mmdebstrap $bootstrap_args \
+                   $arch_param \
+                   --mode=unshare \
+                   ${@get_distro_components_argument(d)} \
+                   "${@get_distro_suite(d)}" \
+                   "${WORKDIR}/rootfs.tar.zst" \
+                   "${@get_distro_source(d)}"
 
     sudo -E -s <<'EOSUDO'
         set -e
-        if [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then
-            arch_param="--arch=${DISTRO_ARCH}"
-        fi
-        ${DEBOOTSTRAP} $debootstrap_args \
-                       $arch_param \
-                       ${@get_distro_components_argument(d)} \
-                       "${@get_distro_suite(d)}" \
-                       "${ROOTFSDIR}" \
-                       "${@get_distro_source(d)}" \
-                       ${DISTRO_DEBOOTSTRAP_SCRIPT}
+
+        tar -xf "${WORKDIR}/rootfs.tar.zst" -C "${ROOTFSDIR}" --exclude="./dev/console"
 
         # Install apt config
         mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d"
@@ -389,7 +394,7 @@ do_bootstrap() {
         umount -l "${ROOTFSDIR}/sys"
         umount -l "${ROOTFSDIR}/base-apt" || true
 
-        # Finalize debootstrap by setting the link in deploy
+        # Finalize bootstrap by setting the link in deploy
         ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
 EOSUDO
     deb_dl_dir_export "${ROOTFSDIR}" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
@@ -406,19 +411,16 @@ SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize"
 
 bootstrap_sstate_prepare() {
     # this runs in SSTATE_BUILDDIR, which will be deleted automatically
-    lopts="--one-file-system --exclude=var/cache/apt/archives"
-    sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar $lopts $(basename "${ROOTFSDIR}")
-    sudo chown $(id -u):$(id -g) bootstrap.tar
+    sudo cp -a "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst" ./bootstrap.tar.zst
+    sudo chown $(id -u):$(id -g) bootstrap.tar.zst
 }
 
 bootstrap_sstate_finalize() {
     # this runs in SSTATE_INSTDIR
-    # - after building the bootstrap, the tar won't be there, but we also don't need to unpack
-    # - after restoring from cache, there will be a tar which we unpack and then delete
-    if [ -f bootstrap.tar ]; then
-        sudo tar -C $(dirname "${ROOTFSDIR}") -xpf bootstrap.tar
-        sudo ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
-        rm bootstrap.tar
+    if [ -f bootstrap.tar.zst ]; then
+        mv bootstrap.tar.zst "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst"
+        sudo ln -Tfsr "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst" \
+                      "${DEPLOY_ISAR_BOOTSTRAP}.tar.zst"
     fi
 }
 
-- 
2.34.1

[PATCH v5 2/7] meta: Preinstall apt for Ubuntu and RPi targets

[Anton Mikanovich]

Apt package is not in required packages in Ubuntu and Raspberry repos.
To have it installed during bootstrap introduce BASE_DISTRO override
and append apt to DISTRO_BOOTSTRAP_BASE_PACKAGES for this distros.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/conf/bitbake.conf                              | 2 +-
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 3 +++
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 80dc01c7..22fdc0e7 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -70,7 +70,7 @@ KERNEL_FILE:riscv64 ?= "vmlinux"
 KERNEL_FILE:arm64 ?= "vmlinux"
 
 MACHINEOVERRIDES ?= "${MACHINE}"
-DISTROOVERRIDES ?= "${DISTRO}"
+DISTROOVERRIDES ?= "${DISTRO}:${BASE_DISTRO}"
 OVERRIDES = "${PACKAGE_ARCH}:${MACHINEOVERRIDES}:${DISTROOVERRIDES}:${BASE_DISTRO_CODENAME}:forcevariable"
 FILESOVERRIDES = "${PACKAGE_ARCH}:${MACHINE}"
 
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 9f1142ba..22fb398f 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -25,6 +25,9 @@ DEPLOY_ISAR_BOOTSTRAP ?= ""
 DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales,usrmerge"
 DISTRO_BOOTSTRAP_BASE_PACKAGES:append:gnupg = ",gnupg"
 DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = ",ca-certificates"
+DISTRO_BOOTSTRAP_BASE_PACKAGES:append:ubuntu = ",apt"
+DISTRO_BOOTSTRAP_BASE_PACKAGES:append:raspios = ",apt"
+DISTRO_BOOTSTRAP_BASE_PACKAGES:append:raspios64 = ",apt"
 DISTRO_VARS_PREFIX ?= "${@'HOST_' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}"
 BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO')}"
 BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}"
-- 
2.34.1

[PATCH v5 3/7] isar-bootstrap: Move preparations to hooks

[Anton Mikanovich]

Move all bootstrap rootfs prepare logic into mmdebstrap hooks.
Also migrate from apt-key to gpg.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 .../isar-bootstrap/isar-bootstrap.inc         | 198 +++++++++---------
 1 file changed, 103 insertions(+), 95 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 22fb398f..8fa22e52 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -213,24 +213,21 @@ def get_distro_components_argument(d):
     else:
         return ""
 
-APT_KEYS_DIR = "${WORKDIR}/aptkeys"
 DISTRO_BOOTSTRAP_KEYRING = "${WORKDIR}/distro-keyring.gpg"
 
-do_generate_keyrings[cleandirs] = "${APT_KEYS_DIR}"
-do_generate_keyrings[dirs] = "${DL_DIR}"
+do_generate_keyrings[cleandirs] = "${WORKDIR}/trusted.gpg.d"
+do_generate_keyrings[dirs] = "${DEBDIR}"
 do_generate_keyrings[vardeps] += "DISTRO_BOOTSTRAP_KEYS THIRD_PARTY_APT_KEYS"
 do_generate_keyrings[network] = "${TASK_USE_SUDO}"
 do_generate_keyrings() {
     if [ -n "${@d.getVar("THIRD_PARTY_APT_KEYFILES") or ""}" ]; then
-        chmod 777 "${APT_KEYS_DIR}"
         for keyfile in ${@d.getVar("THIRD_PARTY_APT_KEYFILES")}; do
-           cp "$keyfile" "${APT_KEYS_DIR}"/"$(basename "$keyfile")"
+           gpg --no-default-keyring --keyring "${DISTRO_BOOTSTRAP_KEYRING}" --import $keyfile
         done
     fi
     if [ -n "${@d.getVar("DISTRO_BOOTSTRAP_KEYFILES") or ""}" ]; then
         for keyfile in ${@d.getVar("DISTRO_BOOTSTRAP_KEYFILES")}; do
-           sudo apt-key --keyring "${DISTRO_BOOTSTRAP_KEYRING}" add $keyfile
-           cp "$keyfile" "${APT_KEYS_DIR}"/"$(basename "$keyfile")"
+           gpg --no-default-keyring --keyring "${DISTRO_BOOTSTRAP_KEYRING}" --import $keyfile
         done
     fi
 }
@@ -271,12 +268,14 @@ do_bootstrap[vardeps] += " \
     ISAR_ENABLE_COMPAT_ARCH \
     ${DISTRO_VARS_PREFIX}DISTRO_APT_SOURCES \
     "
-do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP} ${BOOTSTRAP_TMPDIR}"
+do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP} ${BOOTSTRAP_TMPDIR} ${WORKDIR}/trusted.gpg.d ${WORKDIR}/sources.list.d"
 do_bootstrap[depends] = "base-apt:do_cache isar-apt:do_cache_config"
 do_bootstrap[network] = "${TASK_USE_NETWORK_AND_SUDO}"
 
 inherit compat
 
+DEB_DL_LOCK ?= "${DEBDIR}/${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}.lock"
+
 do_bootstrap() {
     if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then
         if [ -z "${COMPAT_DISTRO_ARCH}" ]; then
@@ -286,124 +285,133 @@ do_bootstrap() {
     bootstrap_args="--verbose --variant=minbase --include=${DISTRO_BOOTSTRAP_BASE_PACKAGES}"
     if [ -f "${DISTRO_BOOTSTRAP_KEYRING}" ]; then
         bootstrap_args="$bootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}"
+        cp "${DISTRO_BOOTSTRAP_KEYRING}" "${WORKDIR}/trusted.gpg.d/"
     fi
     E="${@ isar_export_proxies(d)}"
-    export BOOTSTRAP_FOR_HOST
 
-    deb_dl_dir_import "${ROOTFSDIR}" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
     sudo rm -rf --one-file-system "${ROOTFSDIR}"
     mkdir -p "${ROOTFSDIR}"
 
+    if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
+        base_apt_tmp="$(mktemp -d /tmp/isar-base-aptXXXXXXXXXX)"
+        bootstrap_list="${WORKDIR}/sources.list.d/base-apt.list"
+        line="copy://$base_apt_tmp/${BOOTSTRAP_BASE_DISTRO} ${BASE_DISTRO_CODENAME} main"
+        if [ -z "${BASE_REPO_KEY}" ]; then
+            line="[trusted=yes] ${line}"
+        fi
+        echo "deb ${line}" >  "${WORKDIR}/sources.list.d/base-apt.list"
+        line="copy://$base_apt_tmp/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} main"
+        if [ -z "${BASE_REPO_KEY}" ]; then
+            line="[trusted=yes] ${line}"
+        fi
+        echo "deb-src ${line}" >>  "${WORKDIR}/sources.list.d/base-apt.list"
+
+        # no need to sync /var/cache/apt/archives if base-apt used
+        syncin='echo skip sync-in'
+        syncout='echo skip sync-out'
+        extra_setup="mount --bind '${REPO_BASE_DIR}' $base_apt_tmp"
+        extra_extract="$syncout"
+        # save mmdebstrap tempdir for cleanup
+        extra_essential="mkdir -p \$1/$base_apt_tmp && \
+                         echo \$1 > ${WORKDIR}/mmtmpdir && \
+                         mount --bind '${REPO_BASE_DIR}' \$1/$base_apt_tmp"
+        # replace base-apt mount in tmp with /base-apt mount
+        extra_customize="sed -i \"s|copy://$base_apt_tmp|file:///base-apt|g\" \
+                             \$1/etc/apt/sources.list.d/*.list && \
+                         mkdir -p \$1/base-apt && \
+                         mount --bind '${REPO_BASE_DIR}' \$1/base-apt && \
+                         chroot \$1 apt-get update -y \
+                                -o APT::Update::Error-Mode=any && \
+                         umount \$1/base-apt && \
+                         umount \$1/$base_apt_tmp && rm ${WORKDIR}/mmtmpdir && \
+                         umount $base_apt_tmp && rm -rf --one-file-system $base_apt_tmp"
+    else
+        deb_dl_dir_import "${WORKDIR}/dl_dir" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
+
+        bootstrap_list="${WORKDIR}/sources.list.d/bootstrap.list"
+        install -v -m644 "${APTSRCS}" \
+                         "${WORKDIR}/sources.list.d/bootstrap.list"
+
+        syncin='flock -s ${DEB_DL_LOCK} cp -n --no-preserve=owner \
+                      "${WORKDIR}/dl_dir/var/cache/apt/archives/"*.deb \
+                      "$1/var/cache/apt/archives/" || true'
+        syncout='flock -s ${DEB_DL_LOCK} cp -n --no-preserve=owner \
+                       "$1/var/cache/apt/archives/"*.deb \
+                       "${WORKDIR}/dl_dir/var/cache/apt/archives/"'
+        extra_setup="$syncin"
+        extra_extract="$syncout"
+        # prefetch apt debs because mmdebstrap will clean them on next stage
+        extra_essential='apt-get install apt -y -d \
+                                 -o Dir::State="$1/var/lib/apt" \
+                                 -o Dir::Etc="$1/etc/apt" \
+                                 -o Dir::Cache="$1/var/cache/apt" \
+                                 -o Apt::Architecture="${BOOTSTRAP_DISTRO_ARCH}"'
+        extra_essential="$extra_essential && $syncout"
+        extra_customize="$syncout"
+    fi
+
     if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then
         export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
     fi
 
     arch_param="--arch=${BOOTSTRAP_DISTRO_ARCH},${DISTRO_ARCH}"
+    if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then
+        arch_param="$arch_param,${COMPAT_DISTRO_ARCH}"
+    fi
+
+    # Cleanup mounts if fails
+    trap 'exit 1' INT HUP QUIT TERM ALRM USR1
+    trap '[ -r "${WORKDIR}/mmtmpdir" ] && tmpdir=$(cat "${WORKDIR}/mmtmpdir") \
+                                       && rm "${WORKDIR}/mmtmpdir"; \
+          [ -d "$tmpdir" ] && mountpoint -q $tmpdir/$base_apt_tmp \
+                           && sudo umount $tmpdir/$base_apt_tmp; \
+          [ -d "$tmpdir" ] && mountpoint -q $tmpdir/base-apt \
+                           && sudo umount $tmpdir/base-apt; \
+          [ -d "$tmpdir" ] && sudo rm -rf --one-file-system $tmpdir; \
+          [ -n "$base_apt_tmp" ] && mountpoint -q $base_apt_tmp \
+                                 && sudo umount $base_apt_tmp \
+                                 && rm -rf --one-file-system $base_apt_tmp' EXIT
 
     sudo TMPDIR="${BOOTSTRAP_TMPDIR}" mmdebstrap $bootstrap_args \
                    $arch_param \
                    --mode=unshare \
+                   --setup-hook='mkdir -p "$1/var/cache/apt/archives/"' \
+                   --setup-hook="$extra_setup" \
+                   --setup-hook='upload "${APTPREFS}" /etc/apt/preferences.d/bootstrap' \
+                   --setup-hook='upload "${APTSRCS_INIT}" /etc/apt/sources-list' \
+                   --setup-hook='upload "${WORKDIR}/locale" /etc/locale' \
+                   --setup-hook='mkdir -p "$1/etc/apt/trusted.gpg.d"' \
+                   --setup-hook='sync-in "${WORKDIR}/trusted.gpg.d" /etc/apt/trusted.gpg.d' \
+                   --setup-hook='install -v -m755 "${WORKDIR}/chroot-setup.sh" "$1/chroot-setup.sh"' \
+                   --extract-hook="$extra_extract" \
+                   --essential-hook="$extra_essential" \
+                   --customize-hook="$extra_customize" \
+                   --customize-hook='sed -i "/en_US.UTF-8 UTF-8/s/^#//g" "$1/etc/locale.gen"' \
+                   --customize-hook='chroot "$1" /usr/sbin/locale-gen' \
+                   --customize-hook='chroot "$1" /usr/bin/apt-get -y clean' \
+                   --skip=cleanup/apt \
+                   --skip=download/empty \
                    ${@get_distro_components_argument(d)} \
                    "${@get_distro_suite(d)}" \
                    "${WORKDIR}/rootfs.tar.zst" \
-                   "${@get_distro_source(d)}"
+                   "$bootstrap_list"
 
     sudo -E -s <<'EOSUDO'
         set -e
 
         tar -xf "${WORKDIR}/rootfs.tar.zst" -C "${ROOTFSDIR}" --exclude="./dev/console"
 
-        # Install apt config
-        mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d"
-        install -v -m644 "${APTPREFS}" \
-                         "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap"
-        mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
-        if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
-            line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} ${BASE_DISTRO_CODENAME} main"
-            if [ -z "${BASE_REPO_KEY}" ]; then
-                line="[trusted=yes] ${line}"
-            fi
-            echo "deb ${line}" >  "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
-            line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} main"
-            if [ -z "${BASE_REPO_KEY}" ]; then
-                line="[trusted=yes] ${line}"
-            fi
-            echo "deb-src ${line}" >>  "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
-
-            mkdir -p ${ROOTFSDIR}/base-apt
-            mount --bind ${REPO_BASE_DIR} ${ROOTFSDIR}/base-apt
-        else
-            install -v -m644 "${APTSRCS}" \
-                             "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
-        fi
-        install -v -m644 "${APTSRCS_INIT}" "${ROOTFSDIR}/etc/apt/sources-list"
-        rm -f "${ROOTFSDIR}/etc/apt/sources.list"
-        rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"*
-        find ${APT_KEYS_DIR}/ -type f | while read keyfile
-        do
-            MY_GPGHOME="$(chroot "${ROOTFSDIR}" mktemp -d /tmp/gpghomeXXXXXXXXXX)"
-            echo "Created temporary directory ${MY_GPGHOME} for gpg-agent"
-            export GNUPGHOME="${MY_GPGHOME}"
-            APT_KEY_APPEND="--homedir ${MY_GPGHOME}"
-
-            kfn="$(basename $keyfile)"
-            cp $keyfile "${ROOTFSDIR}/tmp/$kfn"
-            chroot "${ROOTFSDIR}" /usr/bin/gpg-agent --daemon -- /usr/bin/apt-key \
-                --keyring ${THIRD_PARTY_APT_KEYRING} ${APT_KEY_APPEND} add "/tmp/$kfn"
-            rm "${ROOTFSDIR}/tmp/$kfn"
-
-            echo "Removing ${MY_GPGHOME}"
-            rm -rf "${ROOTFSDIR}${MY_GPGHOME}"
-        done
-
-        # Set locale
-        install -v -m644 "${WORKDIR}/locale" "${ROOTFSDIR}/etc/locale"
-
-        sed -i '/en_US.UTF-8 UTF-8/s/^#//g' "${ROOTFSDIR}/etc/locale.gen"
-        chroot "${ROOTFSDIR}" /usr/sbin/locale-gen
-
         # setup chroot
         install -v -m755 "${WORKDIR}/chroot-setup.sh" "${ROOTFSDIR}/chroot-setup.sh"
         "${ROOTFSDIR}/chroot-setup.sh" "setup" "${ROOTFSDIR}"
 
-        # update APT
-        mount -o bind,private /dev ${ROOTFSDIR}/dev
-        mount --bind /dev/pts ${ROOTFSDIR}/dev/pts
-        mount -t tmpfs none "${ROOTFSDIR}/dev/shm"
-        mount -t proc none ${ROOTFSDIR}/proc
-        mount --rbind /sys ${ROOTFSDIR}/sys
-        mount --make-rslave ${ROOTFSDIR}/sys
-
-        export DEBIAN_FRONTEND=noninteractive
-
-        if [ "${BOOTSTRAP_FOR_HOST}" = "1" ]; then
-            chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${DISTRO_ARCH}
-        fi
-
-        if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then
-            chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${COMPAT_DISTRO_ARCH}
-        fi
-
-        chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y \
-                                -o APT::Update::Error-Mode=any
-        chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f
-        chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \
-                                -o Debug::pkgProblemResolver=yes
-
-        umount -l "${ROOTFSDIR}/dev/shm"
-        umount -l "${ROOTFSDIR}/dev/pts"
-        umount -l "${ROOTFSDIR}/dev"
-        umount -l "${ROOTFSDIR}/proc"
-        umount -l "${ROOTFSDIR}/sys"
-        umount -l "${ROOTFSDIR}/base-apt" || true
-
         # Finalize bootstrap by setting the link in deploy
         ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
 EOSUDO
-    deb_dl_dir_export "${ROOTFSDIR}" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
-
-    # Cleanup apt cache
-    sudo -Es chroot "${ROOTFSDIR}" /usr/bin/apt-get -y clean
+    if [ "${ISAR_USE_CACHED_BASE_REPO}" != "1" ]; then
+        deb_dl_dir_export "${WORKDIR}/dl_dir" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
+        sudo rm -rf --one-file-system "${WORKDIR}/dl_dir"
+    fi
 }
 
 addtask bootstrap before do_build after do_generate_keyrings
-- 
2.34.1

[PATCH v5 4/7] isar-bootstrap: Use tar output instead of directory

[Anton Mikanovich]

Do not unpack tar with rootfs content inside bootstrap recipe.
This change can allow to remove sudo usage during the bootstrap later.
Using tar output requires moving chroot prepare step to the later
unpack step.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/rootfs.bbclass                   | 11 +++++----
 .../isar-bootstrap/isar-bootstrap.inc         | 23 ++++---------------
 2 files changed, 12 insertions(+), 22 deletions(-)

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 65574a6c..98e17c3c 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -73,12 +73,15 @@ rootfs_do_qemu() {
     fi
 }
 
-BOOTSTRAP_SRC = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-host_${DISTRO}-${DISTRO_ARCH}"
-BOOTSTRAP_SRC:${ROOTFS_ARCH} = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTFS_ARCH}"
+BOOTSTRAP_SRC = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-host_${DISTRO}-${DISTRO_ARCH}.tar.zst"
+BOOTSTRAP_SRC:${ROOTFS_ARCH} = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTFS_ARCH}.tar.zst"
 
 rootfs_prepare[weight] = "25"
 rootfs_prepare(){
-    sudo cp -Trpfx --reflink=auto '${BOOTSTRAP_SRC}/' '${ROOTFSDIR}'
+    sudo tar -xf "${BOOTSTRAP_SRC}" -C "${ROOTFSDIR}" --exclude="./dev/console"
+
+    # setup chroot
+    sudo "${ROOTFSDIR}/chroot-setup.sh" "setup" "${ROOTFSDIR}"
 }
 
 ROOTFS_CONFIGURE_COMMAND += "rootfs_configure_isar_apt"
@@ -224,7 +227,7 @@ cache_deb_src() {
     # Note: ISAR updates the apt state information(apt-get update) only once during bootstrap and
     # relies on that through out the build. Copy that state information instead of apt-get update
     # which generates a new state from upstream.
-    sudo cp -Trpn --reflink=auto "${BOOTSTRAP_SRC}/var/lib/apt/lists/" "${ROOTFSDIR}/var/lib/apt/lists/"
+    sudo tar -xf "${BOOTSTRAP_SRC}" ./var/lib/apt/lists --one-top-level="${ROOTFSDIR}"
 
     deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME}
     debsrc_download ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME}
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 8fa22e52..dee6cc31 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -15,7 +15,6 @@ PV = "1.0"
 
 BOOTSTRAP_FOR_HOST ?= "0"
 
-ROOTFSDIR = "${WORKDIR}/rootfs"
 APTPREFS = "${WORKDIR}/apt-preferences"
 APTSRCS = "${WORKDIR}/apt-sources"
 APTSRCS_INIT = "${WORKDIR}/apt-sources-init"
@@ -289,9 +288,6 @@ do_bootstrap() {
     fi
     E="${@ isar_export_proxies(d)}"
 
-    sudo rm -rf --one-file-system "${ROOTFSDIR}"
-    mkdir -p "${ROOTFSDIR}"
-
     if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
         base_apt_tmp="$(mktemp -d /tmp/isar-base-aptXXXXXXXXXX)"
         bootstrap_list="${WORKDIR}/sources.list.d/base-apt.list"
@@ -396,18 +392,9 @@ do_bootstrap() {
                    "${WORKDIR}/rootfs.tar.zst" \
                    "$bootstrap_list"
 
-    sudo -E -s <<'EOSUDO'
-        set -e
-
-        tar -xf "${WORKDIR}/rootfs.tar.zst" -C "${ROOTFSDIR}" --exclude="./dev/console"
-
-        # setup chroot
-        install -v -m755 "${WORKDIR}/chroot-setup.sh" "${ROOTFSDIR}/chroot-setup.sh"
-        "${ROOTFSDIR}/chroot-setup.sh" "setup" "${ROOTFSDIR}"
+    # Finalize bootstrap by setting the link in deploy
+    sudo ln -Tfsr "${WORKDIR}/rootfs.tar.zst" "${DEPLOY_ISAR_BOOTSTRAP}.tar.zst"
 
-        # Finalize bootstrap by setting the link in deploy
-        ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
-EOSUDO
     if [ "${ISAR_USE_CACHED_BASE_REPO}" != "1" ]; then
         deb_dl_dir_export "${WORKDIR}/dl_dir" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
         sudo rm -rf --one-file-system "${WORKDIR}/dl_dir"
@@ -422,15 +409,15 @@ SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize"
 
 bootstrap_sstate_prepare() {
     # this runs in SSTATE_BUILDDIR, which will be deleted automatically
-    sudo cp -a "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst" ./bootstrap.tar.zst
+    sudo cp -a "${WORKDIR}/rootfs.tar.zst" ./bootstrap.tar.zst
     sudo chown $(id -u):$(id -g) bootstrap.tar.zst
 }
 
 bootstrap_sstate_finalize() {
     # this runs in SSTATE_INSTDIR
     if [ -f bootstrap.tar.zst ]; then
-        mv bootstrap.tar.zst "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst"
-        sudo ln -Tfsr "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst" \
+        mv bootstrap.tar.zst "${WORKDIR}/rootfs.tar.zst"
+        sudo ln -Tfsr "${WORKDIR}/rootfs.tar.zst" \
                       "${DEPLOY_ISAR_BOOTSTRAP}.tar.zst"
     fi
 }
-- 
2.34.1

[PATCH v5 5/7] user_manual.md: Replace debootstrap with mmdebstrap

[Anton Mikanovich]

Update documentation in user manual.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 doc/user_manual.md | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index ecb987f4..6cbff8cf 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -67,7 +67,9 @@ Install the following packages:
 apt install \
   binfmt-support \
   bzip2 \
-  debootstrap \
+  mmdebstrap \
+  arch-test \
+  apt-utils \
   dpkg-dev \
   gettext-base \
   git \
@@ -127,7 +129,7 @@ apt install qemu
 
 ### Setup Sudo
 
-Isar requires `sudo` rights without password to work with `chroot` and `debootstrap`. To add them, use the following steps:
+Isar requires `sudo` rights without password to work with `chroot`. To add them, use the following steps:
 ```
  # visudo
 ```
-- 
2.34.1

[PATCH v5 6/7] user_manual.md: Update minimal host requirements

[Anton Mikanovich]

Using mmdebstrap requires at least Debian Bullseye on host.
Update user manual with the new minimal host distro.
Also remove all Buster-related notes.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 doc/user_manual.md | 35 ++++++++++++-----------------------
 1 file changed, 12 insertions(+), 23 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 6cbff8cf..512721c9 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -48,7 +48,7 @@ For demonstration purposes, Isar provides support for the following
 configurations:
 
  - QEMU ARM with Debian Buster
- - QEMU ARM64 with Debian Buster (for host >= buster)
+ - QEMU ARM64 with Debian Buster
  - QEMU i386 with Debian Buster
  - QEMU amd64 with Debian Buster
  - Raspberry Pi various models with Raspberry OS Bullseye
@@ -60,7 +60,7 @@ The steps below describe how to build the images provided by default.
 
 ### Install Host Tools
 
-The supported host system is >= buster.
+The minimal supported host system is Debian Bullseye.
 
 Install the following packages:
 ```
@@ -74,6 +74,7 @@ apt install \
   gettext-base \
   git \
   python3 \
+  python3-distutils \
   quilt \
   qemu-user-static \
   reprepro \
@@ -87,28 +88,19 @@ apt install \
   zstd
 ```
 
-If your host is >= buster, also install the following package.
-```
-apt install python3-distutils
-```
-
-**NOTE:** sbuild version (<=0.78.1) packaged in Debian Buster doesn't support
-`$apt_keep_downloaded_packages` option which is required in Isar for
-populating `${DL_DIR}/deb`. So, host `sbuild` in this case should be manually
-upgraded to >=0.81.2 version from Debian Bullseye.
-
 Next, the user who should run Isar needs to be added to `sbuild` group.
 ```
 sudo gpasswd -a <username> sbuild
 ```
 
 If you want to generate containerized SDKs, also install the following 
-packages: `umoci` and `skopeo`.
-Umoci is provided by Debian Buster and can be installed with 
-`apt install umoci`, Skopeo is provided by Debian Bullseye/Unstable and has to 
-be installed either manually downloading the DEB and installing it (no other 
-packages required) or with `apt install -t bullseye skopeo` (if 
-unstable/bullseye included in `/etc/apt/sources.list[.d]`).
+packages:
+
+```
+apt install \
+  umoci \
+  skopeo
+```
 
 Notes:
 
@@ -234,9 +226,6 @@ Variables may be used in `.wks.in` files; Isar will expand them and generate a r
 In order to run the EFI images with `qemu`, an EFI firmware is required and available at the following address:
 https://github.com/tianocore/edk2/tree/3858b4a1ff09d3243fea8d07bd135478237cb8f7
 
-Note that the `ovmf` package in Debian Buster contains a pre-compiled firmware, but doesn't seem to be recent
-enough to allow images to be testable under `qemu`.
-
 ```
 # AMD64 image, EFI
 qemu-system-x86_64 -m 256M -nographic -bios edk2/Build/OvmfX64/RELEASE_*/FV/OVMF.fd -hda tmp/deploy/images/qemuamd64/isar-image-base-debian-buster-qemuamd64.wic
@@ -963,8 +952,8 @@ put into Isar apt.
 Debian cross-compilation works out of the box. Currently the following build configurations are supported in Isar:
 
  - buster armhf
- - buster arm64 (for host >= buster)
- - buster mipsel (for host >= buster)
+ - buster arm64
+ - buster mipsel
  - bullseye armhf
  - bullseye arm64
  - bullseye mipsel
-- 
2.34.1

[PATCH v5 7/7] isar-bootstrap: Remove unused code

[Anton Mikanovich]

Some external logic like installing gpg and ca-certificates is now
doing by mmdebstrap inself. So remove duplications.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 .../isar-bootstrap/isar-bootstrap.inc         | 26 -------------------
 1 file changed, 26 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index dee6cc31..33bdf6b9 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -22,8 +22,6 @@ DISTRO_BOOTSTRAP_KEYFILES = ""
 THIRD_PARTY_APT_KEYFILES = ""
 DEPLOY_ISAR_BOOTSTRAP ?= ""
 DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales,usrmerge"
-DISTRO_BOOTSTRAP_BASE_PACKAGES:append:gnupg = ",gnupg"
-DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = ",ca-certificates"
 DISTRO_BOOTSTRAP_BASE_PACKAGES:append:ubuntu = ",apt"
 DISTRO_BOOTSTRAP_BASE_PACKAGES:append:raspios = ",apt"
 DISTRO_BOOTSTRAP_BASE_PACKAGES:append:raspios64 = ",apt"
@@ -178,30 +176,6 @@ def get_distro_primary_source_entry(d):
             return source[2:]
     bb.fatal('Invalid apt sources list')
 
-def get_distro_have_https_source(d):
-    return any(source[2].startswith("https://") for source in generate_distro_sources(d))
-
-def get_distro_needs_https_support(d):
-    if get_distro_have_https_source(d):
-        return "https-support"
-    else:
-        return ""
-
-OVERRIDES:append = ":${@get_distro_needs_https_support(d)}"
-
-def get_distro_needs_gpg_support(d):
-    if d.getVar("DISTRO_BOOTSTRAP_KEYS") or \
-       d.getVar("THIRD_PARTY_APT_KEYS") or \
-       d.getVar("BASE_REPO_KEY"):
-        return "gnupg"
-    else:
-        return ""
-
-OVERRIDES:append = ":${@get_distro_needs_gpg_support(d)}"
-
-def get_distro_source(d):
-    return get_distro_primary_source_entry(d)[0]
-
 def get_distro_suite(d):
     return get_distro_primary_source_entry(d)[1]
 
-- 
2.34.1

Re: [PATCH v5 4/7] isar-bootstrap: Use tar output instead of directory

[Anton Mikanovich]

Hello,

Just some statistics to proof zst compression choosing.
Building mc:qemuarm64-bookworm:isar-image-base with 3 types of bootstrap tar
compression gives the following measurements:

       | bootstrap, sec | chroot1, sec | chroot2, sec
  gz |          73           |         80        |         88
  xz |          76           |         82        |         89
zst |          68           |         80        |         88

where:
bootstrap - do_bootstrap task of isar-bootstrap-target
chroot1 - do_rootfs_install task of sbuild-chroot-target
chroot2 - do_rootfs_install task of sbuild-chroot-target-db2m
(chroot1 and chroot2 are performing in parallel)