Re: [PATCH v3 0/7] Migrate to mmdebstrap

["Schaffner, Tobias" <tobias.schaffner@siemens.com>]

[-- Attachment #1.1: Type: text/plain, Size: 698 bytes --]

Hi Uladzimir,

some of the patches of this story needed a rebase. Find them attached.

On 20.06.23 12:00, Uladzimir Bely wrote:
...
>>
>> Is this now settled? Asking as I'm planning to release kas 4.0 with
>> the
>> dependencies in patch 7 resolved these days, and I'd like to see this
>> targeting the final version of this feature.
>>
>> Jan
>>
> 
> We are currently upgrading our interlnal CI and need to run more jobs
> before merging it.
> 
> But anyway, if mmdebstrap (and other packages mentioned in patch 3) is
> preinstalled in kas 4.0 image, it would be nice.

The dependencies needed are already included in KAS 4.0.

Is there something else missing to finish this?

All the best,
Tobias

[-- Attachment #1.2: 0005-user_manual-md-Update-minimal-host-requirements.patch --]
[-- Type: text/x-patch, Size: 3439 bytes --]

From: Anton Mikanovich <amikan@ilbers.de>
Subject: [PATCH v3 5/7] user_manual.md: Update minimal host requirements
Date: Wed, 31 May 2023 13:11:00 +0300
Content-Type: text/plain; charset="UTF-8"

Using mmdebstrap requires at least Debian Bullseye on host.
Update user manual with the new minimal host distro.
Also remove all Buster-related notes.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 doc/user_manual.md | 35 ++++++++++++-----------------------
 1 file changed, 12 insertions(+), 23 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index dc7ed4c8..72dd49e3 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -48,7 +48,7 @@ For demonstration purposes, Isar provides support for the following
 configurations:
 
  - QEMU ARM with Debian Buster
- - QEMU ARM64 with Debian Buster (for host >= buster)
+ - QEMU ARM64 with Debian Buster
  - QEMU i386 with Debian Buster
  - QEMU amd64 with Debian Buster
  - Raspberry Pi various models with Raspberry OS Bullseye
@@ -60,7 +60,7 @@ The steps below describe how to build the images provided by default.
 
 ### Install Host Tools
 
-The supported host system is >= buster.
+The minimal supported host system is Debian Bullseye.
 
 Install the following packages:
 ```
@@ -74,6 +74,7 @@ apt install \
   gettext-base \
   git \
   python3 \
+  python3-distutils \
   quilt \
   qemu-user-static \
   reprepro \
@@ -87,28 +88,13 @@ apt install \
   zstd
 ```
 
-If your host is >= buster, also install the following package.
-```
-apt install python3-distutils
-```
-
-**NOTE:** sbuild version (<=0.78.1) packaged in Debian Buster doesn't support
-`$apt_keep_downloaded_packages` option which is required in Isar for
-populating `${DL_DIR}/deb`. So, host `sbuild` in this case should be manually
-upgraded to >=0.81.2 version from Debian Bullseye.
-
 Next, the user who should run Isar needs to be added to `sbuild` group.
 ```
 sudo gpasswd -a <username> sbuild
 ```
 
 If you want to generate containerized SDKs, also install the following 
-packages: `umoci` and `skopeo`.
-Umoci is provided by Debian Buster and can be installed with 
-`apt install umoci`, Skopeo is provided by Debian Bullseye/Unstable and has to 
-be installed either manually downloading the DEB and installing it (no other 
-packages required) or with `apt install -t bullseye skopeo` (if 
-unstable/bullseye included in `/etc/apt/sources.list[.d]`).
+packages:
 
 Notes:
 
@@ -234,9 +220,6 @@ Variables may be used in `.wks.in` files; Isar will expand them and generate a r
 In order to run the EFI images with `qemu`, an EFI firmware is required and available at the following address:
 https://github.com/tianocore/edk2/tree/3858b4a1ff09d3243fea8d07bd135478237cb8f7
 
-Note that the `ovmf` package in Debian Buster contains a pre-compiled firmware, but doesn't seem to be recent
-enough to allow images to be testable under `qemu`.
-
 ```
 # AMD64 image, EFI
 qemu-system-x86_64 -m 256M -nographic -bios edk2/Build/OvmfX64/RELEASE_*/FV/OVMF.fd -hda tmp/deploy/images/qemuamd64/isar-image-base-debian-buster-qemuamd64.wic
@@ -963,8 +946,8 @@ put into Isar apt.
 Debian cross-compilation works out of the box. Currently the following build configurations are supported in Isar:
 
  - buster armhf
- - buster arm64 (for host >= buster)
- - buster mipsel (for host >= buster)
+ - buster arm64
+ - buster mipsel
  - bullseye armhf
  - bullseye arm64
  - bullseye mipsel
-- 
2.34.1

[-- Attachment #1.3: 0004-user_manual-md-Replace-debootstrap-with-mmdebstrap.patch --]
[-- Type: text/x-patch, Size: 1029 bytes --]

From: Anton Mikanovich <amikan@ilbers.de>
Subject: [PATCH v3 4/7] user_manual.md: Replace debootstrap with mmdebstrap
Date: Wed, 31 May 2023 13:10:59 +0300
Content-Type: text/plain; charset="UTF-8"

Update documentation in user manual.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 doc/user_manual.md | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index be47fdcc..dc7ed4c8 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -67,7 +67,9 @@ Install the following packages:
 apt install \
   binfmt-support \
   bzip2 \
-  debootstrap \
+  mmdebstrap \
+  arch-test \
+  apt-utils \
   dpkg-dev \
   gettext-base \
   git \
@@ -127,7 +129,7 @@ apt install qemu
 
 ### Setup Sudo
 
-Isar requires `sudo` rights without password to work with `chroot` and `debootstrap`. To add them, use the following steps:
+Isar requires `sudo` rights without password to work with `chroot`. To add them, use the following steps:
 ```
  # visudo
 ```
-- 
2.34.1

[-- Attachment #1.4: 0001-meta-Switch-to-mmdebstrap.patch --]
[-- Type: text/x-patch, Size: 6108 bytes --]

From: Anton Mikanovich <amikan@ilbers.de>
Subject: [PATCH v3 1/7] meta: Switch to mmdebstrap
Date: Wed, 31 May 2023 13:10:56 +0300
Content-Type: text/plain; charset="UTF-8"

From: Uladzimir Bely <ubely@ilbers.de>

Instead of debootstrap, use mmdebstrap alternative.

Internally, it uses apt and allows to debootstrap the distro from
multiple repositories.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta-isar/conf/distro/ubuntu-common.inc       |  4 --
 .../isar-bootstrap/isar-bootstrap.inc         | 57 ++++++++++---------
 2 files changed, 30 insertions(+), 31 deletions(-)

diff --git a/meta-isar/conf/distro/ubuntu-common.inc b/meta-isar/conf/distro/ubuntu-common.inc
index 6a990a89..9ca1273d 100644
--- a/meta-isar/conf/distro/ubuntu-common.inc
+++ b/meta-isar/conf/distro/ubuntu-common.inc
@@ -19,9 +19,6 @@ HOST_DISTRO_BOOTSTRAP_KEYS += "${BOOTSTRAP_KEY}"
 # kernel package name is linux-image-generic in Ubuntu
 KERNEL_NAME = "generic"
 
-# that is what debootstrap_1.0.118ubuntu1 does anyways
-DISTRO_DEBOOTSTRAP_SCRIPT = "/usr/share/debootstrap/scripts/gutsy"
-
 DISTRO_CONFIG_SCRIPT ?= "debian-configscript.sh"
 
 DISTRO_KERNELS ?= " \

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index b94ae0bd..f89c284e 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -15,7 +15,7 @@ PV = "1.0"
 
 BOOTSTRAP_FOR_HOST ?= "0"
 
-DEBOOTSTRAP ?= "qemu-debootstrap"
+DEBOOTSTRAP = "mmdebstrap"
 ROOTFSDIR = "${WORKDIR}/rootfs"
 APTPREFS = "${WORKDIR}/apt-preferences"
 APTSRCS = "${WORKDIR}/apt-sources"
@@ -23,12 +23,14 @@ APTSRCS_INIT = "${WORKDIR}/apt-sources-init"
 DISTRO_BOOTSTRAP_KEYFILES = ""
 THIRD_PARTY_APT_KEYFILES = ""
 DEPLOY_ISAR_BOOTSTRAP ?= ""
-DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"
+DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales,usrmerge"
 DISTRO_BOOTSTRAP_BASE_PACKAGES:append:gnupg = ",gnupg"
 DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = ",ca-certificates"
 DISTRO_VARS_PREFIX ?= "${@'HOST_' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}"
 BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO')}"
 BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}"
+BOOTSTRAP_DISTRO_ARCH = "${@d.getVar('HOST_ARCH' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO_ARCH')}"
+BOOTSTRAP_TMPDIR = "${WORKDIR}/tempdir"
 FILESEXTRAPATHS:append = ":${BBPATH}"
 
 inherit deb-dl-dir
@@ -270,7 +272,7 @@ do_bootstrap[vardeps] += " \
     ISAR_ENABLE_COMPAT_ARCH \
     ${DISTRO_VARS_PREFIX}DISTRO_APT_SOURCES \
     "
-do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP}"
+do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP} ${BOOTSTRAP_TMPDIR}"
 do_bootstrap[depends] = "base-apt:do_cache isar-apt:do_cache_config"
 do_bootstrap[network] = "${TASK_USE_NETWORK_AND_SUDO}"
 
@@ -286,27 +288,31 @@ do_bootstrap() {
     if [ -f "${DISTRO_BOOTSTRAP_KEYRING}" ]; then
         debootstrap_args="$debootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}"
     fi
-    if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" -a -z "${BASE_REPO_KEY}" ]; then
-        debootstrap_args="$debootstrap_args --no-check-gpg"
-    fi
     E="${@ isar_export_proxies(d)}"
-    export BOOTSTRAP_FOR_HOST debootstrap_args E
+    export BOOTSTRAP_FOR_HOST
 
-    sudo rm -rf --one-file-system "${ROOTFSDIR}"
     deb_dl_dir_import "${ROOTFSDIR}" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
+    sudo rm -rf --one-file-system "${ROOTFSDIR}"
+    mkdir -p "${ROOTFSDIR}"
+
+    if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then
+        export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
+    fi
+
+    arch_param="--arch=${BOOTSTRAP_DISTRO_ARCH},${DISTRO_ARCH}"
+
+    sudo TMPDIR="${BOOTSTRAP_TMPDIR}" ${DEBOOTSTRAP} $debootstrap_args \
+                   $arch_param \
+                   --mode=unshare \
+                   ${@get_distro_components_argument(d)} \
+                   "${@get_distro_suite(d)}" \
+                   "${WORKDIR}/rootfs.tar.zst" \
+                   "${@get_distro_source(d)}"
 
     sudo -E -s <<'EOSUDO'
         set -e
-        if [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then
-            arch_param="--arch=${DISTRO_ARCH}"
-        fi
-        ${DEBOOTSTRAP} $debootstrap_args \
-                       $arch_param \
-                       ${@get_distro_components_argument(d)} \
-                       "${@get_distro_suite(d)}" \
-                       "${ROOTFSDIR}" \
-                       "${@get_distro_source(d)}" \
-                       ${DISTRO_DEBOOTSTRAP_SCRIPT}
+
+        tar -xf "${WORKDIR}/rootfs.tar.zst" -C "${ROOTFSDIR}" --exclude="./dev/console"
 
         # Install apt config
         mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d"
@@ -409,19 +415,16 @@ SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize"
 
 bootstrap_sstate_prepare() {
     # this runs in SSTATE_BUILDDIR, which will be deleted automatically
-    lopts="--one-file-system --exclude=var/cache/apt/archives"
-    sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar $lopts $(basename "${ROOTFSDIR}")
-    sudo chown $(id -u):$(id -g) bootstrap.tar
+    sudo cp -a "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst" ./bootstrap.tar.zst
+    sudo chown $(id -u):$(id -g) bootstrap.tar.zst
 }
 
 bootstrap_sstate_finalize() {
     # this runs in SSTATE_INSTDIR
-    # - after building the bootstrap, the tar won't be there, but we also don't need to unpack
-    # - after restoring from cache, there will be a tar which we unpack and then delete
-    if [ -f bootstrap.tar ]; then
-        sudo tar -C $(dirname "${ROOTFSDIR}") -xpf bootstrap.tar
-        sudo ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
-        rm bootstrap.tar
+    if [ -f bootstrap.tar.zst ]; then
+        mv bootstrap.tar.zst "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst"
+        sudo ln -Tfsr "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst" \
+                      "${DEPLOY_ISAR_BOOTSTRAP}.tar.zst"
     fi
 }
 
-- 
2.34.1

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 10139 bytes --]