[PATCH v2 0/4] Imager schroot migration

[PATCH v2 0/4] Imager schroot migration

[Anton Mikanovich]

This patchset moves Isar imager from buildchroot to schroot usage as the next
step after moving to sbuild.

Imagers of every target are now running in separate schroot sessions with
independent overlays on top of basic unchanged schroot. It means we need to
open schroot session before any imager task execution and close this session
after the usage. Any changes made inside overlay will be dropped after the
usage to keep basic schroot clean.

We also need to introduce additional cleanup inside isar-events finish handler
to prevent leaving sessions in case we execute only session start task.

There are still couple of issues need to be solved before the usage:
- there are no documentaion updates
- only basic build scenarios works, CI still fails

Changes since v1:

- Rebased on next and bitbake update
- Fix some build cases

Anton Mikanovich (4):
  sbuild: Allow setting custom config paths
  imager: Migrate from buildchroot to schroot
  imager: Move image types to schroot
  events: Cleanup lost schroot sessions if any

 meta/classes/image-tools-extension.bbclass | 80 ++++++++++++++++---
 meta/classes/image.bbclass                 |  4 +-
 meta/classes/imagetypes_vm.bbclass         |  7 +-
 meta/classes/imagetypes_wic.bbclass        | 89 ++++++++--------------
 meta/classes/isar-events.bbclass           | 16 ++++
 meta/classes/sbuild.bbclass                | 51 +++++++++----
 6 files changed, 159 insertions(+), 88 deletions(-)

-- 
2.17.1

[PATCH v2 1/4] sbuild: Allow setting custom config paths

[Anton Mikanovich]

Extend internal schroot APIs to be able to set custom chroot config
name. This will allow to configure/unconfigure schroots from any task.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/sbuild.bbclass | 51 +++++++++++++++++++++++++++----------
 1 file changed, 38 insertions(+), 13 deletions(-)

diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 6e3c790c..06c01d67 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -31,10 +31,18 @@ SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
 
 schroot_create_configs() {
     mkdir -p "${TMPDIR}/schroot-overlay"
-    sudo -s <<'EOSUDO'
+
+    conf_dir="${SBUILD_CONF_DIR}"
+    conf_file="${SCHROOT_CONF_FILE}"
+    if [ -n "$1" ]; then
+        conf_dir="${SCHROOT_CONF}/${1}"
+        conf_file="${SCHROOT_CONF}/chroot.d/${1}"
+    fi
+    export conf_dir conf_file
+    sudo -E -s <<'EOSUDO'
         set -e
 
-        cat << EOF > "${SCHROOT_CONF_FILE}"
+        cat << EOF > "${conf_file}"
 [${SBUILD_CHROOT}]
 type=directory
 directory=${SCHROOT_DIR}
@@ -51,8 +59,8 @@ preserve-environment=true
 EOF
 
         # Prepare mount points
-        cp -rf "${SCHROOT_CONF}/sbuild" "${SBUILD_CONF_DIR}"
-        sbuild_fstab="${SBUILD_CONF_DIR}/fstab"
+        cp -rf "${SCHROOT_CONF}/sbuild" "${conf_dir}"
+        sbuild_fstab="${conf_dir}/fstab"
 
         fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind 0 0"
         grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}
@@ -68,12 +76,19 @@ EOSUDO
 }
 
 schroot_delete_configs() {
-    sudo -s <<'EOSUDO'
+    conf_dir="${SBUILD_CONF_DIR}"
+    conf_file="${SCHROOT_CONF_FILE}"
+    if [ -n "$1" ]; then
+        conf_dir="${SCHROOT_CONF}/${1}"
+        conf_file="${SCHROOT_CONF}/chroot.d/${1}"
+    fi
+    export conf_dir conf_file
+    sudo -E -s <<'EOSUDO'
         set -e
-        if [ -d "${SBUILD_CONF_DIR}" ]; then
-            rm -rf "${SBUILD_CONF_DIR}"
+        if [ -d "${conf_dir}" ]; then
+            rm -rf "${conf_dir}"
         fi
-        rm -f "${SCHROOT_CONF_FILE}"
+        rm -f "${conf_file}"
 EOSUDO
 }
 
@@ -107,22 +122,32 @@ sbuild_export() {
 }
 
 insert_mounts() {
-    sudo -s <<'EOSUDO'
+    conf_dir="${SBUILD_CONF_DIR}"
+    if [ -n "$1" ]; then
+        conf_dir="${SCHROOT_CONF}/${1}"
+    fi
+    export conf_dir
+    sudo -E -s <<'EOSUDO'
         set -e
         for mp in ${SCHROOT_MOUNTS}; do
             FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
-            grep -qxF "${FSTAB_LINE}" ${SBUILD_CONF_DIR}/fstab || \
-                echo "${FSTAB_LINE}" >> ${SBUILD_CONF_DIR}/fstab
+            grep -qxF "${FSTAB_LINE}" ${conf_dir}/fstab || \
+                echo "${FSTAB_LINE}" >> ${conf_dir}/fstab
         done
 EOSUDO
 }
 
 remove_mounts() {
-    sudo -s <<'EOSUDO'
+    conf_dir="${SBUILD_CONF_DIR}"
+    if [ -n "$1" ]; then
+        conf_dir="${SCHROOT_CONF}/${1}"
+    fi
+    export conf_dir
+    sudo -E -s <<'EOSUDO'
         set -e
         for mp in ${SCHROOT_MOUNTS}; do
             FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
-            sed -i "\|${FSTAB_LINE}|d" ${SBUILD_CONF_DIR}/fstab
+            sed -i "\|${FSTAB_LINE}|d" ${conf_dir}/fstab
         done
 EOSUDO
 }
-- 
2.17.1

[PATCH v2 2/4] imager: Migrate from buildchroot to schroot

[Anton Mikanovich]

Install dependencies and perform all imager actions using one schroot
overlay. This requires to open a session which stays opened until all
the images are ready. This session will be closed then and all the
changes will be lost.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/image-tools-extension.bbclass | 80 +++++++++++++++++++---
 meta/classes/image.bbclass                 |  1 +
 2 files changed, 72 insertions(+), 9 deletions(-)

diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass
index 2d3dda4f..85f0d551 100644
--- a/meta/classes/image-tools-extension.bbclass
+++ b/meta/classes/image-tools-extension.bbclass
@@ -5,13 +5,15 @@
 #
 # This file extends the image.bbclass to supply tools for futher imager functions
 
-inherit buildchroot
+inherit sbuild
 
 IMAGER_INSTALL ??= ""
 IMAGER_BUILD_DEPS ??= ""
 DEPENDS += "${IMAGER_BUILD_DEPS}"
 
-do_install_imager_deps[depends] = "${BUILDCHROOT_DEP} isar-apt:do_cache_config"
+IMAGER_SCHROOT_SESSION_ID = "isar-imager-${SCHROOT_USER}-${PN}-${MACHINE}-${ISAR_BUILD_UUID}"
+
+do_install_imager_deps[depends] = "${SCHROOT_DEP} isar-apt:do_cache_config"
 do_install_imager_deps[deptask] = "do_deploy_deb"
 do_install_imager_deps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
 do_install_imager_deps[network] = "${TASK_USE_NETWORK_AND_SUDO}"
@@ -25,11 +27,9 @@ do_install_imager_deps() {
         distro="${HOST_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
     fi
 
-    buildchroot_do_mounts
-
     E="${@ isar_export_proxies(d)}"
-    deb_dl_dir_import ${BUILDCHROOT_DIR} ${distro}
-    sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
+    deb_dl_dir_import ${SCHROOT_DIR} ${distro}
+    schroot -r -c ${IMAGER_SCHROOT_SESSION_ID} -d / -u root -- sh -c ' \
         apt-get update \
             -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
             -o Dir::Etc::SourceParts="-" \
@@ -38,10 +38,72 @@ do_install_imager_deps() {
             --allow-unauthenticated --allow-downgrades --download-only install \
             ${IMAGER_INSTALL}'
 
-    deb_dl_dir_export ${BUILDCHROOT_DIR} ${distro}
-    sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
+    deb_dl_dir_export ${SCHROOT_DIR} ${distro}
+    schroot -r -c ${IMAGER_SCHROOT_SESSION_ID} -d / -u root -- sh -c ' \
         apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y \
             --allow-unauthenticated --allow-downgrades install \
             ${IMAGER_INSTALL}'
 }
-addtask install_imager_deps before do_image_tools
+addtask install_imager_deps before do_image_tools after do_start_imager_session
+
+SCHROOT_MOUNTS = "${WORKDIR}:${PP_WORK} ${IMAGE_ROOTFS}:${PP_ROOTFS} ${DEPLOY_DIR_IMAGE}:${PP_DEPLOY}"
+
+do_start_imager_session[dirs] = "${WORKDIR} ${IMAGE_ROOTFS} ${DEPLOY_DIR_IMAGE}"
+do_start_imager_session[depends] = "${SCHROOT_DEP} isar-apt:do_cache_config"
+do_start_imager_session[nostamp] = "1"
+do_start_imager_session[network] = "${TASK_USE_SUDO}"
+python do_start_imager_session() {
+    import subprocess
+    bb.build.exec_func("schroot_create_configs", d)
+    bb.build.exec_func("insert_mounts", d)
+    sbuild_chroot = d.getVar("SBUILD_CHROOT", True)
+    session_id = d.getVar("IMAGER_SCHROOT_SESSION_ID", True)
+    try:
+        if subprocess.run("schroot -l --all-sessions | grep %s" % session_id, shell=True).returncode:
+            subprocess.run("schroot -b -c %s -n %s" % (sbuild_chroot, session_id), shell=True, check=True)
+            bb.debug(2, "Open schroot session %s" % session_id)
+        else:
+            subprocess.run("schroot --recover-session -c %s" % session_id, shell=True, check=True)
+            bb.debug(2, "Reuse schroot session %s" % session_id)
+        d.setVar("SCHROOT_OPEN_SESSION_ID", session_id)
+    except subprocess.CalledProcessError as err:
+        subprocess.run("schroot -e -c %s" % session_id, shell=True)
+        bb.build.exec_func("remove_mounts", d)
+        bb.build.exec_func("schroot_delete_configs", d)
+        bb.fatal("Could not create schroot session: %s" % err.output.decode('utf-8') if err.output else "")
+}
+addtask start_imager_session before do_stop_imager_session after do_rootfs_finalize
+
+do_stop_imager_session[depends] = "${SCHROOT_DEP}"
+do_stop_imager_session[nostamp] = "1"
+do_stop_imager_session[network] = "${TASK_USE_SUDO}"
+python do_stop_imager_session() {
+    import subprocess
+    session_id = d.getVar("IMAGER_SCHROOT_SESSION_ID", True)
+    try:
+        id = subprocess.run("schroot -d / -r -c %s -- printenv -0 SCHROOT_ALIAS_NAME" % session_id,
+                            shell=True, check=True, stdout=subprocess.PIPE).stdout.decode('utf-8')
+        bb.debug(2, "Close schroot session %s (%s)" % (session_id, id))
+        subprocess.run("schroot -e -c %s" % session_id, shell=True, check=True)
+    except subprocess.CalledProcessError as err:
+        bb.error("Could not close schroot session %s: %s" % (session_id, err.output.decode('utf-8')) if err.output else "")
+    finally:
+        if 'id' in locals():
+            d.setVar("SBUILD_CHROOT", id)
+            bb.build.exec_func("remove_mounts", d)
+            bb.build.exec_func("schroot_delete_configs", d)
+}
+addtask stop_imager_session before do_deploy after do_image
+
+imager_run() {
+    imager_cleanup() {
+        if id="$(schroot -d / -r -c ${IMAGER_SCHROOT_SESSION_ID} -- printenv -0 SCHROOT_ALIAS_NAME)"; then
+            schroot -e -c ${IMAGER_SCHROOT_SESSION_ID}
+            remove_mounts $id
+            schroot_delete_configs $id
+        fi
+    }
+    trap 'exit 1' INT HUP QUIT TERM ALRM USR1
+    trap 'imager_cleanup' EXIT
+    schroot -r -c ${IMAGER_SCHROOT_SESSION_ID} "$@"
+}
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index ba0fd6ef..5851b94d 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -296,6 +296,7 @@ python() {
         d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps))
         d.appendVarFlag(task, 'vardepsexclude', ' ' + ' '.join(vardepsexclude))
         d.appendVarFlag(task, 'dirs', localdata.expand(' ${DEPLOY_DIR_IMAGE}'))
+        d.appendVarFlag(task, 'deptask', ' do_start_imager_session')
         if task_deps:
             d.appendVarFlag(task, 'depends', task_deps)
         bb.build.addtask(task, 'do_image', after, d)
-- 
2.17.1

[PATCH v2 3/4] imager: Move image types to schroot

[Anton Mikanovich]

Use schroot inside imager session to prepare images.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/image.bbclass          |  3 +-
 meta/classes/imagetypes_vm.bbclass  |  7 +--
 meta/classes/imagetypes_wic.bbclass | 89 ++++++++++-------------------
 3 files changed, 33 insertions(+), 66 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 5851b94d..4e58bb4b 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -157,7 +157,7 @@ inherit ${IMGCLASSES}
 # convenience variables to be used by CMDs
 IMAGE_FILE_HOST = "${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.${type}"
 IMAGE_FILE_CHROOT = "${PP_DEPLOY}/${IMAGE_FULLNAME}.${type}"
-SUDO_CHROOT = "sudo chroot ${BUILDCHROOT_DIR}"
+SUDO_CHROOT = "imager_run -d ${PP_ROOTFS} -u root --"
 
 # hook up IMAGE_CMD_*
 python() {
@@ -229,7 +229,6 @@ python() {
             imager_build_deps.add(dep)
 
         # construct image command
-        cmds.append('\timage_do_mounts')
         image_cmd = localdata.getVar('IMAGE_CMD:' + bt_clean)
         if image_cmd:
             localdata.setVar('type', bt)
diff --git a/meta/classes/imagetypes_vm.bbclass b/meta/classes/imagetypes_vm.bbclass
index 81ef866f..67ab63b5 100644
--- a/meta/classes/imagetypes_vm.bbclass
+++ b/meta/classes/imagetypes_vm.bbclass
@@ -4,8 +4,6 @@
 # This class allows to generate images for VMware and VirtualBox
 #
 
-inherit buildchroot
-
 USING_OVA = "${@bb.utils.contains('IMAGE_BASETYPES', 'ova', '1', '0', d)}"
 
 FILESEXTRAPATHS:prepend := "${LAYERDIR_core}/classes/vm-img:"
@@ -38,9 +36,8 @@ CONVERSION_OPTIONS = "${@set_convert_options(d)}"
 
 convert_wic() {
     rm -f '${DEPLOY_DIR_IMAGE}/${VIRTUAL_MACHINE_IMAGE_FILE}'
-    image_do_mounts
     bbnote "Creating ${VIRTUAL_MACHINE_IMAGE_FILE} from ${SOURCE_IMAGE_FILE}"
-    sudo -E  chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+    imager_run -p -d ${PP_WORK} -- \
     /usr/bin/qemu-img convert -f raw -O ${VIRTUAL_MACHINE_IMAGE_TYPE} ${CONVERSION_OPTIONS} \
         '${PP_DEPLOY}/${SOURCE_IMAGE_FILE}' '${VIRTUAL_MACHINE_DISK}'
 }
@@ -83,7 +80,7 @@ IMAGE_CMD:ova() {
     export LAST_CHANGE=$(date -u "+%Y-%m-%dT%H:%M:%SZ")
     export OVA_FIRMWARE_UPPERCASE=$(echo ${OVA_FIRMWARE} | tr '[a-z]' '[A-Z]')
     export OVF_TEMPLATE_STAGE2=$(echo ${OVF_TEMPLATE_FILE} | sed 's/.tmpl$//' )
-    sudo -Es chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} <<'EOSUDO'
+    imager_run -p -d ${PP_WORK} <<'EOSUDO'
         set -e
         export DISK_SIZE_BYTES=$(qemu-img info -f vmdk "${VIRTUAL_MACHINE_DISK}" \
                                  | gawk 'match($0, /^virtual size:.*\(([0-9]+) bytes\)/, a) {print a[1]}')
diff --git a/meta/classes/imagetypes_wic.bbclass b/meta/classes/imagetypes_wic.bbclass
index 5f0df484..74c4604b 100644
--- a/meta/classes/imagetypes_wic.bbclass
+++ b/meta/classes/imagetypes_wic.bbclass
@@ -83,8 +83,6 @@ python () {
             bb.build.addtask('do_transform_template', 'do_image_wic', None, d)
 }
 
-inherit buildchroot
-
 IMAGER_INSTALL:wic += "${WIC_IMAGER_INSTALL}"
 # wic comes with reasonable defaults, and the proper interface is the wks file
 ROOTFS_EXTRA ?= "0"
@@ -95,7 +93,6 @@ STAGING_DIR ?= "${TMPDIR}"
 IMAGE_BASENAME ?= "${PN}-${DISTRO}"
 FAKEROOTCMD ?= "${SCRIPTSDIR}/wic_fakeroot"
 RECIPE_SYSROOT_NATIVE ?= "/"
-BUILDCHROOT_DIR = "${BUILDCHROOT_TARGET_DIR}"
 
 WIC_CREATE_EXTRA_ARGS ?= ""
 WIC_DEPLOY_PARTITIONS ?= "0"
@@ -145,26 +142,11 @@ check_for_wic_warnings() {
 
 do_image_wic[file-checksums] += "${WKS_FILE_CHECKSUM}"
 IMAGE_CMD:wic() {
-    wic_do_mounts
     generate_wic_image
     check_for_wic_warnings
 }
 
-wic_do_mounts[vardepsexclude] += "BITBAKEDIR"
-wic_do_mounts() {
-    buildchroot_do_mounts
-    sudo -s <<'EOSUDO'
-        ( flock 9
-        set -e
-        for dir in ${BBLAYERS} ${STAGING_DIR} ${SCRIPTSDIR} ${BITBAKEDIR}; do
-            mkdir -p ${BUILDCHROOT_DIR}/$dir
-            if ! mountpoint ${BUILDCHROOT_DIR}/$dir >/dev/null 2>&1; then
-                mount --bind --make-private $dir ${BUILDCHROOT_DIR}/$dir
-            fi
-        done
-        ) 9>${MOUNT_LOCKFILE}
-EOSUDO
-}
+SCHROOT_MOUNTS += "${BBLAYERS} ${STAGING_DIR} ${SCRIPTSDIR} ${BITBAKEDIR}"
 
 generate_wic_image[vardepsexclude] += "WKS_FULL_PATH BITBAKEDIR TOPDIR"
 generate_wic_image() {
@@ -174,50 +156,39 @@ generate_wic_image() {
     mkdir -p ${IMAGE_ROOTFS}/../pseudo
     touch ${IMAGE_ROOTFS}/../pseudo/files.db
 
-    # create the temp dir in the buildchroot to ensure uniqueness
-    WICTMP=$(cd ${BUILDCHROOT_DIR}; mktemp -d -p tmp)
-
-    sudo -E chroot ${BUILDCHROOT_DIR} \
-        sh -c ' \
-          BITBAKEDIR="$1"
-          SCRIPTSDIR="$2"
-          WKS_FULL_PATH="$3"
-          STAGING_DIR="$4"
-          MACHINE="$5"
-          WICTMP="$6"
-          IMAGE_FULLNAME="$7"
-          IMAGE_BASENAME="$8"
-          shift 8
-          # The python path is hard-coded as /usr/bin/python3-native/python3 in wic. Handle that.
-          mkdir -p /usr/bin/python3-native/
-          if [ $(head -1 $(which bmaptool) | grep python3) ];then
+    imager_run -p -d ${PP_WORK} -u root <<'EOSUDO'
+        set -e
+
+        # The python path is hard-coded as /usr/bin/python3-native/python3 in wic. Handle that.
+        mkdir -p /usr/bin/python3-native/
+        if [ $(head -1 $(which bmaptool) | grep python3) ];then
             ln -fs /usr/bin/python3 /usr/bin/python3-native/python3
-          else
+        else
             ln -fs /usr/bin/python2 /usr/bin/python3-native/python3
-          fi
-          export PATH="$BITBAKEDIR/bin:$PATH"
-          "$SCRIPTSDIR"/wic create "$WKS_FULL_PATH" \
-            --vars "$STAGING_DIR/$MACHINE/imgdata/" \
-            -o "/$WICTMP/${IMAGE_FULLNAME}.wic/" \
+        fi
+
+        export PATH="${BITBAKEDIR}/bin:$PATH"
+
+        "${SCRIPTSDIR}"/wic create "${WKS_FULL_PATH}" \
+            --vars "${STAGING_DIR}/${MACHINE}/imgdata/" \
+            -o "/tmp/${IMAGE_FULLNAME}.wic/" \
             --bmap \
-            -e "$IMAGE_BASENAME" $@' \
-              my_script "${BITBAKEDIR}" "${SCRIPTSDIR}" "${WKS_FULL_PATH}" "${STAGING_DIR}" \
-              "${MACHINE}" "${WICTMP}" "${IMAGE_FULLNAME}" "${IMAGE_BASENAME}" \
-              ${WIC_CREATE_EXTRA_ARGS}
+            -e "${IMAGE_BASENAME}" ${WIC_CREATE_EXTRA_ARGS}
+
+        WIC_DIRECT=$(ls -t -1 /tmp/${IMAGE_FULLNAME}.wic/*.direct | head -1)
+        mv -f ${WIC_DIRECT} ${PP_DEPLOY}/${IMAGE_FULLNAME}.wic
+        mv -f ${WIC_DIRECT}.bmap ${PP_DEPLOY}/${IMAGE_FULLNAME}.wic.bmap
+        # deploy partition files if requested (ending with .p<x>)
+        if [ "${WIC_DEPLOY_PARTITIONS}" -eq "1" ]; then
+            # locate *.direct.p<x> partition files
+            find "/tmp/${IMAGE_FULLNAME}.wic/" -type f -regextype sed -regex ".*\.direct.*\.p[0-9]\{1,\}" | while read f; do
+                suffix=$(basename $f | sed 's/.*\.direct\(.*\)/\1/')
+                mv -f ${f} ${PP_DEPLOY}/${IMAGE_FULLNAME}.wic${suffix}
+            done
+        fi
+EOSUDO
 
     sudo chown -R $(stat -c "%U" ${LAYERDIR_core}) ${LAYERDIR_core} ${LAYERDIR_isar} ${SCRIPTSDIR} || true
-    WIC_DIRECT=$(ls -t -1 ${BUILDCHROOT_DIR}/$WICTMP/${IMAGE_FULLNAME}.wic/*.direct | head -1)
-    sudo chown -R $(id -u):$(id -g) ${BUILDCHROOT_DIR}/${WICTMP}
-    mv -f ${WIC_DIRECT} ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic
-    mv -f ${WIC_DIRECT}.bmap ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic.bmap
-    # deploy partition files if requested (ending with .p<x>)
-    if [ "${WIC_DEPLOY_PARTITIONS}" -eq "1" ]; then
-        # locate *.direct.p<x> partition files
-        find ${BUILDCHROOT_DIR}/${WICTMP} -type f -regextype sed -regex ".*\.direct.*\.p[0-9]\{1,\}" | while read f; do
-            suffix=$(basename $f | sed 's/.*\.direct\(.*\)/\1/')
-            mv -f ${f} ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic${suffix}
-        done
-    fi
-    rm -rf ${BUILDCHROOT_DIR}/${WICTMP}
+    sudo chown -R $(id -u):$(id -g) ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic*
     rm -rf ${IMAGE_ROOTFS}/../pseudo
 }
-- 
2.17.1

[PATCH v2 4/4] events: Cleanup lost schroot sessions if any

[Anton Mikanovich]

In case the user will only open imager schroot session but not close it
we need to find and clean it up.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/isar-events.bbclass | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass
index f52b2349..5343d4b1 100644
--- a/meta/classes/isar-events.bbclass
+++ b/meta/classes/isar-events.bbclass
@@ -51,6 +51,22 @@ python build_completed() {
 
     basepath = tmpdir + '/work/'
 
+    build_uuid = d.getVar('ISAR_BUILD_UUID', True)
+    sessions = subprocess.run('schroot -l --all-sessions | grep isar | grep %s' % build_uuid,
+                              shell=True, stdout=subprocess.PIPE).stdout.decode('utf-8')
+    for line in sessions.splitlines():
+        session_id = line.split(':', 1)[1]
+        bb.debug(1, 'Closing imager session %s' % session_id)
+        id = subprocess.run("schroot -d / -r -c %s -- printenv -0 SCHROOT_ALIAS_NAME" % session_id,
+                            shell=True, check=True, stdout=subprocess.PIPE).stdout.decode('utf-8')
+        if id:
+            subprocess.run('schroot --recover-session -c %s' % session_id, shell=True, check=True)
+            subprocess.run('schroot -e -c %s' % session_id, shell=True, check=True)
+        if 'id' in locals():
+            d.setVar('SBUILD_CHROOT', id)
+            bb.build.exec_func('remove_mounts', d)
+            bb.build.exec_func('schroot_delete_configs', d)
+
     with open('/proc/mounts') as f:
         for line in f.readlines():
             if basepath in line:
-- 
2.17.1

Re: [PATCH v2 0/4] Imager schroot migration

[Anton Mikanovich]

30.12.2022 22:08, Anton Mikanovich wrote:
> This patchset moves Isar imager from buildchroot to schroot usage as the next
> step after moving to sbuild.
>
> Imagers of every target are now running in separate schroot sessions with
> independent overlays on top of basic unchanged schroot. It means we need to
> open schroot session before any imager task execution and close this session
> after the usage. Any changes made inside overlay will be dropped after the
> usage to keep basic schroot clean.
>
> We also need to introduce additional cleanup inside isar-events finish handler
> to prevent leaving sessions in case we execute only session start task.
>
> There are still couple of issues need to be solved before the usage:
> - there are no documentaion updates
> - only basic build scenarios works, CI still fails
>
> Changes since v1:
>
> - Rebased on next and bitbake update
> - Fix some build cases

There are a lot of work still need to be done here.
I will be out of office for the next 2 weeks, will back 16.01.23
Feel free to propose any ideas or fixes for this patchset in the meantime.

Re: [PATCH v2 4/4] events: Cleanup lost schroot sessions if any

[Roberto A. Foglietta]

[-- Attachment #1: Type: text/plain, Size: 1334 bytes --]

On Fri, 30 Dec 2022 at 20:08, Anton Mikanovich <amikan@ilbers.de> wrote:
>
> In case the user will only open imager schroot session but not close it
> we need to find and clean it up.

Hi Anton,

 this patchset works for me but it missing a 5th patch to free dpkg
base class from buildchroot, here in attachment.
 The patch has been also sent by elastic mail SMTP and it went in
moderation as usual.

>From 79e97dd3fe5fa2e710ca48dd864fd3df1bdb5a3c Mon Sep 17 00:00:00 2001
From: "Roberto A. Foglietta" <roberto.foglietta@gmail.com>
Date: Mon, 2 Jan 2023 14:46:21 +0100
Subject: [PATCH] dpkg base class: use schroot only not buildchroot anymore

The patchset v.2 by Anton Mikanovich to migrate buildchroot to schroot is not
complete because the buildchroot is still used by the dpkg base class which can
be freed by buildchroot with this patch which applies on the top of these four:

    * events: Cleanup lost schroot sessions if any, v2
    * imager: Move image types to schroot, v2
    * imager: Migrate from buildchroot to schroot, v2
    * sbuild: Allow setting custom config paths, v2

Signed-off-by: Roberto A. Foglietta <roberto.foglietta@gmail.com>
---
 meta/classes/dpkg-base.bbclass | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

https://github.com/ilbers/isar/commit/79e97dd3fe5fa2e710ca48dd864fd3df1bdb5a3c

[-- Attachment #2: 0001-dpkg-base-class-use-schroot-only-not-buildchroot-any.patch --]
[-- Type: text/x-patch, Size: 1767 bytes --]

From 79e97dd3fe5fa2e710ca48dd864fd3df1bdb5a3c Mon Sep 17 00:00:00 2001
From: "Roberto A. Foglietta" <roberto.foglietta@gmail.com>
Date: Mon, 2 Jan 2023 14:46:21 +0100
Subject: [PATCH] dpkg base class: use schroot only not buildchroot anymore

The patchset v.2 by Anton Mikanovich to migrate buildchroot to schroot is not
complete because the buildchroot is still used by the dpkg base class which can
be freed by buildchroot with this patch which applies on the top of these four:

    * events: Cleanup lost schroot sessions if any, v2
    * imager: Move image types to schroot, v2
    * imager: Migrate from buildchroot to schroot, v2
    * sbuild: Allow setting custom config paths, v2

Signed-off-by: Roberto A. Foglietta <roberto.foglietta@gmail.com>
---
 meta/classes/dpkg-base.bbclass | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 260aa73..b0035f5 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -5,7 +5,7 @@
 # SPDX-License-Identifier: MIT
 
 inherit sbuild
-inherit buildchroot
+#inherit buildchroot
 inherit debianize
 inherit terminal
 inherit repository
@@ -124,7 +124,7 @@ addtask apt_fetch
 do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
 
 # Add dependency from the correct buildchroot: host or target
-do_apt_fetch[depends] += "${BUILDCHROOT_DEP}"
+#do_apt_fetch[depends] += "${BUILDCHROOT_DEP}"
 
 # Add dependency from the correct schroot: host or target
 do_apt_fetch[depends] += "${SCHROOT_DEP}"
@@ -194,7 +194,7 @@ dpkg_do_mounts() {
     mkdir -p ${BUILDROOT}
     sudo mount --bind ${WORKDIR} ${BUILDROOT}
 
-    buildchroot_do_mounts
+#   buildchroot_do_mounts
 }
 
 dpkg_undo_mounts() {
-- 
2.34.1

Re: [PATCH v2 4/4] events: Cleanup lost schroot sessions if any

[Roberto A. Foglietta]

On Tue, 3 Jan 2023 at 10:55, Roberto A. Foglietta
<roberto.foglietta@gmail.com> wrote:
>
> On Fri, 30 Dec 2022 at 20:08, Anton Mikanovich <amikan@ilbers.de> wrote:
> >
> > In case the user will only open imager schroot session but not close it
> > we need to find and clean it up.
>
> Hi Anton,
>
>  this patchset works for me but it missing a 5th patch to free dpkg
> base class from buildchroot, here in attachment.
>  The patch has been also sent by elastic mail SMTP and it went in
> moderation as usual.

Hi Anton,

I have identified an unexpected behaviour - for me an improvement but
technically speaking a regression - after having applied your patchset
about schroot migration.

Try this:

git clone https://github.com/robang74/isar-nvidia-debian.git
cd isar-nvidia-debian
put c9f4b21 in kas:

These patches are useful/necessary to let my project building:

* c9f4b21 - image tools ext. class defines ISAR_CROSS_COMPILE, if
missing  (HEAD)
* c945877 - dpkg class set default value for ISAR_CROSS_COMPILE, if missing
* adafe11 - dpkg class sbuild allows extra arguments by vars

This is my patch to migrate the dpkg base class to schroot:

* 423c2f3 - dpkg base class: use schroot only not buildchroot anymore

These are your patches for schroot migration:

* e2d2165 - events: Cleanup lost schroot sessions if any, v2
* 3fb634e - imager: Move image types to schroot, v2
* 51b08db - imager: Migrate from buildchroot to schroot, v2
* 0fea6b1 - sbuild: Allow setting custom config paths, v2

Then do:

source .profile
clean all
build basic-os
build

The second build should do nothing. Instead, it builds the wic image
again starting from image deps: 'do_install_imager_deps'. In my
opinion this is a feature because when I use 'wicshell' and I chroot
into the image to modify it, sometimes I break things and I want to be
back to the original. This allows me to avoid cleaning the build and
restart it again which takes a lot of time compared to starting again
from image deps. However, from a technical point of view it is a bug.

I hope this helps, R-