[PATCH v10 00/19] Sbuild/Schroot migration

[PATCH v10 00/19] Sbuild/Schroot migration

[Anton Mikanovich]

This is a patchset showing how sbuild/schroot tools can be integrated
into Isar build system.

Base schroot images are created with sbuild-chroot-target or, in case
of cross-build, sbuild-chroot-host recipes. These images are based on
isar-bootstrap ones, but include some build-related stuff preinstalled.

To use these changes you need to have sbuild and schroot installed and
configured. User who runs build should be added to sbuild group.

Currently, sbuild-related stuff is supported in latest kas.

If Gilab CI is used, the following configuration changes are required:
- external dir for schroot overlay should be set in the config
/etc/gitlab-runner/config.toml:

volumes = ["/path/to/overlay:/var/lib/schroot/union/overlay"]

The patchset was tested with 'meta-iot2050' and 'xenomai-images'
downstreams in combination with kas image mentioned above.

Current limitations:
- parallel building of several packages requires more free space in
comparison with buildchroot-based architecture. This happens due
the sbuild architecture which uses some basic layer (common for all
packages) and per-package separate layer (where builddeps are downloaded
and installed).

Changes sinse v9:
- Add caching improvement patch;
- Remove per-package deb artifacts dir after usage to reduce disk space;
- Improve schroot cleanup on failures.

Changes since v8:
- Added patch with support of package versions with epochs

Changes since v7:
- fixed work with proxy by adding some environment filters in sbuild;
- DEB_BUILD_PROFILES support simplified and improved
- Fixed gbp-based recipes rebuilding (get back --git-ignore-new option
previously removed)
- fixed base-apt caching for packages and source packages temporary
installed in sbuild isolated environment
- fixed rare build error related to ccache setup (added locking)

Changes since v6:
- added patch that switches sbuild from using source directory to using
.dsc file. The file is generated before sbuild call by `dpkg-source`.

Changes since v5:
- warning about shell exports in dpkg_runbuild_prepend. This should
avoided be reworked to templates.

Changes since v4:
- consider shell exports done in dpkg_runbuild_prepend and pass them
to sbuild environment;
- fixed ccache work.

Changes since v3:
- dpkg_do_mounts() and dpkg_undo_mounts() are not removed for downstreams
compatibility;
- dpkg_build_export is used for adjusting sbuild environment;
- DEB_BUILD_PROFILES support for sbuild is fixed in case of cross-build.

Changes since v2:
- patches reworked/squashed for easier reading and understanding;
- fixed building foreigh architectures with kas-docker;
- implemented support of ccache;
- fixed devshell and devshell_nodeps.

Changes since v1:
- parallel builds with different BUILD_DIR are supported;
- parallel multiconfig targets in one build are supported;
- per-task schroot configuration in /etc/schroot/ is now used;
- patchset now passes Jenkins CI (so patches changes RFC => PATCH).
*** BLURB HERE ***

Anton Mikanovich (2):
  dpkg: Build packages with sbuild
  dpkg-base: Cleanup on schroot fail

Felix Moessbauer (3):
  fix: support build of packages with epoch version
  always create apt-cache dirs in deb_dl_dir_import
  avoid absolute SCHROOT_* paths to improve caching

Uladzimir Bely (14):
  dpkg-gbp: Use separate command to export tarball
  dpkg-gbp: Use host tools for dsc preparation
  sbuild: Add recipes for host and target rootfs to run sbuild
  sbuild: Introduce a class for another build method
  sbuild: Support of DEB_BUILD_PROFILES
  sbuild: Support of shell exports from dpkg_runbuild_prepend
  dpkg: Remove builddeps install task
  sbuild: Add ccache support
  dpkg-base: Switch devshell to use schroot
  dpkg-base: Switch apt_fetch and apt_unpack to use schroot
  doc: Add sbuild-related documentation
  sbuild: Use .dsc file instead of source directory
  sbuild: Fixed proxy support
  sbuild: Fix debsrc_download for packages dependencies

 doc/user_manual.md                            |  22 ++-
 meta/classes/deb-dl-dir.bbclass               |   6 +-
 meta/classes/dpkg-base.bbclass                |  88 ++++++----
 meta/classes/dpkg-gbp.bbclass                 |  26 +--
 meta/classes/dpkg.bbclass                     | 109 +++++++++---
 meta/classes/sbuild.bbclass                   | 159 ++++++++++++++++++
 meta/conf/bitbake.conf                        |   2 +
 .../sbuild-chroot/sbuild-chroot-host.bb       |  13 ++
 .../sbuild-chroot/sbuild-chroot-target.bb     |  10 ++
 .../sbuild-chroot/sbuild-chroot.inc           |  39 +++++
 10 files changed, 394 insertions(+), 80 deletions(-)
 create mode 100644 meta/classes/sbuild.bbclass
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc

-- 
2.17.1

[PATCH v10 01/19] dpkg-gbp: Use separate command to export tarball

[Anton Mikanovich]

From: Uladzimir Bely <ubely@ilbers.de>

We don't actually build the package with gbp, but only prepare it for
the building with external dpkg-buildpackage command. In case there is
no need in real build we can perform only export which will produce
upstream tarball to pass for later building.
This allows to use any build tool for later processing.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-gbp.bbclass | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index d956e8c..6129d73 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -30,5 +30,9 @@ do_install_builddeps_append() {
 }
 
 dpkg_runbuild_prepend() {
-    export GBP_PREFIX="gbp buildpackage --git-ignore-new ${GBP_EXTRA_OPTIONS} --git-builder="
+    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+        sh -c "cd ${PP}/${PPS} && gbp buildpackage --git-ignore-new --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}"
+    # NOTE: `buildpackage --git-builder=/bin/true --git-pristine-tar` is used
+    # for compatibility with gbp version froms debian-stretch. In newer distros
+    # it's possible to use a subcommand `export-orig --pristine-tar`
 }
-- 
2.17.1

[PATCH v10 02/19] dpkg-gbp: Use host tools for dsc preparation

[Anton Mikanovich]

From: Uladzimir Bely <ubely@ilbers.de>

Instead of preinstalling gbp-related packages in sbuild chroot
we will use their host versions in order to keep sbuild environment
as clean as possible.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 doc/user_manual.md            |  4 +++-
 meta/classes/dpkg-gbp.bbclass | 24 ++++--------------------
 2 files changed, 7 insertions(+), 21 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index cdb7322..bcff8d9 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -81,7 +81,9 @@ apt install \
   qemu \
   qemu-user-static \
   reprepro \
-  sudo
+  sudo \
+  git-buildpackage \
+  pristine-tar
 ```
 
 If your host is >= buster, also install the following package.
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index 6129d73..2b0b818 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -9,29 +9,13 @@ S = "${WORKDIR}/git"
 
 PATCHTOOL ?= "git"
 
-GBP_DEPENDS ?= "git-buildpackage pristine-tar"
 GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
 
-do_install_builddeps_append() {
-    dpkg_do_mounts
-    distro="${DISTRO}"
-    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
-       distro="${HOST_DISTRO}"
-    fi
-    deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} \
-        apt-get install -y -o Debug::pkgProblemResolver=yes \
-                        --no-install-recommends --download-only ${GBP_DEPENDS}
-    deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} \
-        apt-get install -y -o Debug::pkgProblemResolver=yes \
-                        --no-install-recommends ${GBP_DEPENDS}
-    dpkg_undo_mounts
-}
-
 dpkg_runbuild_prepend() {
-    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
-        sh -c "cd ${PP}/${PPS} && gbp buildpackage --git-ignore-new --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}"
+    sh -c "
+        cd ${WORKDIR}/${PPS}
+        gbp buildpackage --git-ignore-new --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}
+    "
     # NOTE: `buildpackage --git-builder=/bin/true --git-pristine-tar` is used
     # for compatibility with gbp version froms debian-stretch. In newer distros
     # it's possible to use a subcommand `export-orig --pristine-tar`
-- 
2.17.1

[PATCH v10 03/19] sbuild: Add recipes for host and target rootfs to run sbuild

[Anton Mikanovich]

From: Uladzimir Bely <ubely@ilbers.de>

Similar to buildchroot, we need a separate rootfs to be used
for schroot.

It's based on bootstrapped rootfs, but includes several
common build-related packages.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 .../sbuild-chroot/sbuild-chroot-host.bb       | 13 +++++++
 .../sbuild-chroot/sbuild-chroot-target.bb     | 10 ++++++
 .../sbuild-chroot/sbuild-chroot.inc           | 36 +++++++++++++++++++
 3 files changed, 59 insertions(+)
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc

diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
new file mode 100644
index 0000000..aa82846
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
@@ -0,0 +1,13 @@
+# Root filesystem for packages building
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2021 ilbers GmbH
+
+DESCRIPTION = "Isar sbuild/schroot filesystem for host"
+
+SBUILD_VARIANT = "host"
+
+require sbuild-chroot.inc
+
+ROOTFS_ARCH = "${HOST_ARCH}"
+ROOTFS_DISTRO = "${HOST_DISTRO}"
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
new file mode 100644
index 0000000..d75d783
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
@@ -0,0 +1,10 @@
+# Root filesystem for packages building
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2021 ilbers GmbH
+
+DESCRIPTION = "Isar sbuild/schroot filesystem for target"
+
+SBUILD_VARIANT = "target"
+
+require sbuild-chroot.inc
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
new file mode 100644
index 0000000..177a8a6
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -0,0 +1,36 @@
+# Common part for build chroot filesystem.
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2021 ilbers GmbH
+
+LICENSE = "gpl-2.0"
+LIC_FILES_CHKSUM = "file://${LAYERDIR_core}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
+
+PV = "1.0"
+
+inherit rootfs
+
+SBUILD_CHROOT_PREINSTALL_COMMON = " \
+    fakeroot \
+    build-essential \
+    debhelper \
+"
+
+SBUILD_CHROOT_PREINSTALL ?= " \
+    ${SBUILD_CHROOT_PREINSTALL_COMMON} \
+"
+
+SBUILD_CHROOT_DIR = "${WORKDIR}/rootfs"
+ROOTFSDIR = "${SBUILD_CHROOT_DIR}"
+ROOTFS_PACKAGES = "${SBUILD_CHROOT_PREINSTALL}"
+
+# We don't need /etc/apt/sources.list.d/isar-apt.list' while it's handled by sbuild
+ROOTFS_CONFIGURE_COMMAND_remove = "rootfs_configure_isar_apt"
+
+DEPLOY_SCHROOT = "${@d.getVar('SCHROOT_' + d.getVar('SBUILD_VARIANT').upper() + '_DIR')}"
+
+do_sbuildchroot_deploy[dirs] = "${DEPLOY_DIR}/schroot-${SBUILD_VARIANT}"
+do_sbuildchroot_deploy() {
+    ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_SCHROOT}"
+}
+addtask sbuildchroot_deploy before do_build after do_rootfs
-- 
2.17.1

[PATCH v10 04/19] sbuild: Introduce a class for another build method

[Anton Mikanovich]

From: Uladzimir Bely <ubely@ilbers.de>

This also adds mounts for base-apt inside schroot and adds import/export
deb files to/from schroot. So that it becomes possible to run second
 `cached` build from local base-apt repo.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/sbuild.bbclass | 108 ++++++++++++++++++++++++++++++++++++
 meta/conf/bitbake.conf      |   2 +
 2 files changed, 110 insertions(+)
 create mode 100644 meta/classes/sbuild.bbclass

diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
new file mode 100644
index 0000000..b9fc6a0
--- /dev/null
+++ b/meta/classes/sbuild.bbclass
@@ -0,0 +1,108 @@
+# This software is a part of ISAR.
+# Copyright (C) 2021 ilbers GmbH
+
+SCHROOT_CONF ?= "/etc/schroot"
+
+SCHROOT_MOUNTS ?= ""
+
+python __anonymous() {
+    import pwd
+    d.setVar('SCHROOT_USER', pwd.getpwuid(os.geteuid()).pw_name)
+    d.setVar('SCHROOT_USER_HOME', pwd.getpwuid(os.geteuid()).pw_dir)
+
+    mode = d.getVar('ISAR_CROSS_COMPILE', True)
+    distro_arch = d.getVar('DISTRO_ARCH')
+    if mode == "0" or d.getVar('HOST_ARCH') ==  distro_arch or \
+       (d.getVar('HOST_DISTRO') == "debian-stretch" and distro_arch == "i386"):
+        d.setVar('SBUILD_HOST_ARCH', distro_arch)
+        d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_TARGET_DIR'))
+        dep = "sbuild-chroot-target:do_build"
+    else:
+        d.setVar('SBUILD_HOST_ARCH', d.getVar('HOST_ARCH'))
+        d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_HOST_DIR'))
+        dep = "sbuild-chroot-host:do_build"
+    d.setVar('SCHROOT_DEP', dep)
+}
+
+SBUILD_CHROOT ?= "${DEBDISTRONAME}-${SCHROOT_USER}-${@os.getpid()}"
+
+SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}"
+SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}"
+
+schroot_create_configs() {
+    sudo -s <<'EOSUDO'
+        set -e
+
+        cat << EOF > "${SCHROOT_CONF_FILE}"
+[${SBUILD_CHROOT}]
+type=directory
+directory=${SCHROOT_DIR}
+profile=${SBUILD_CHROOT}
+users=${SCHROOT_USER}
+groups=root,sbuild
+root-users=${SCHROOT_USER}
+root-groups=root,sbuild
+source-root-users=${SCHROOT_USER}
+source-root-groups=root,sbuild
+union-type=overlay
+preserve-environment=true
+EOF
+
+        # Prepare mount points
+        cp -rf "${SCHROOT_CONF}/sbuild" "${SBUILD_CONF_DIR}"
+        sbuild_fstab="${SBUILD_CONF_DIR}/fstab"
+
+        fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind 0 0"
+        grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}
+
+        if [ -d ${DL_DIR} ]; then
+            fstab_downloads="${DL_DIR} /downloads none rw,bind 0 0"
+            grep -qxF "${fstab_downloads}" ${sbuild_fstab} || echo "${fstab_downloads}" >> ${sbuild_fstab}
+        fi
+EOSUDO
+}
+
+schroot_delete_configs() {
+    sudo -s <<'EOSUDO'
+        set -e
+        if [ -d "${SBUILD_CONF_DIR}" ]; then
+            rm -rf "${SBUILD_CONF_DIR}"
+        fi
+        rm -f "${SCHROOT_CONF_FILE}"
+EOSUDO
+}
+
+sbuild_export() {
+    SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
+    VAR=${1}; shift
+    VAR_LINE="'${VAR}' => '${@}',"
+    if [ -s "${SBUILD_CONFIG}" ]; then
+        sed -i -e "\$i\\" -e "${VAR_LINE}" ${SBUILD_CONFIG}
+    else
+        echo "\$build_environment = {" > ${SBUILD_CONFIG}
+        echo "${VAR_LINE}" >> ${SBUILD_CONFIG}
+        echo "};" >> ${SBUILD_CONFIG}
+    fi
+    export SBUILD_CONFIG="${SBUILD_CONFIG}"
+}
+
+insert_mounts() {
+    sudo -s <<'EOSUDO'
+        set -e
+        for mp in ${SCHROOT_MOUNTS}; do
+            FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
+            grep -qxF "${FSTAB_LINE}" ${SBUILD_CONF_DIR}/fstab || \
+                echo "${FSTAB_LINE}" >> ${SBUILD_CONF_DIR}/fstab
+        done
+EOSUDO
+}
+
+remove_mounts() {
+    sudo -s <<'EOSUDO'
+        set -e
+        for mp in ${SCHROOT_MOUNTS}; do
+            FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
+            sed -i "\|${FSTAB_LINE}|d" ${SBUILD_CONF_DIR}/fstab
+        done
+EOSUDO
+}
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index db1180a..07f199d 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -58,6 +58,8 @@ SSTATE_DIR ?= "${TOPDIR}/sstate-cache"
 SSTATE_MANIFESTS = "${TMPDIR}/sstate-control/${DISTRO}-${DISTRO_ARCH}"
 BUILDCHROOT_HOST_DIR = "${DEPLOY_DIR_BUILDCHROOT}-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
 BUILDCHROOT_TARGET_DIR = "${DEPLOY_DIR_BUILDCHROOT}-target/${DISTRO}-${DISTRO_ARCH}"
+SCHROOT_HOST_DIR = "${DEPLOY_DIR}/schroot-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
+SCHROOT_TARGET_DIR = "${DEPLOY_DIR}/schroot-target/${DISTRO}-${DISTRO_ARCH}"
 SDKCHROOT_DIR = "${DEPLOY_DIR_SDKCHROOT}/${DISTRO}-${DISTRO_ARCH}"
 CACHE = "${TMPDIR}/cache"
 KERNEL_FILE ?= "vmlinuz"
-- 
2.17.1

[PATCH v10 05/19] dpkg: Build packages with sbuild

[Anton Mikanovich]

Use previously exported to schroot rootfs to build the package with
sbuild tool. Provide isar-apt as extra repository to be automatically
add to apt sources.

Also added /home/.git-downloads mount while it's used
as git alternates location.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 31 ++++++++++++++++---------------
 meta/classes/dpkg-gbp.bbclass  |  2 ++
 meta/classes/dpkg.bbclass      | 10 ++++++++--
 meta/classes/sbuild.bbclass    |  4 ++--
 4 files changed, 28 insertions(+), 19 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 86933c5..c60052d 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -4,6 +4,7 @@
 #
 # SPDX-License-Identifier: MIT
 
+inherit sbuild
 inherit buildchroot
 inherit debianize
 inherit terminal
@@ -16,6 +17,8 @@ DEPENDS_append_riscv64 = "${@' crossbuild-essential-riscv64' if d.getVar('ISAR_C
 DEB_BUILD_PROFILES ?= ""
 DEB_BUILD_OPTIONS ?= ""
 
+ISAR_APT_REPO ?= "deb [trusted=yes] file:///home/builder/${PN}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO} ${DEBDISTRONAME} main"
+
 python do_adjust_git() {
     import subprocess
 
@@ -177,6 +180,7 @@ addtask prepare_build after do_patch do_transform_template before do_dpkg_build
 # If Isar recipes depend on each other, they typically need the package
 # deployed to isar-apt
 do_prepare_build[deptask] = "do_deploy_deb"
+do_prepare_build[depends] = "${SCHROOT_DEP}"
 
 BUILDROOT = "${BUILDCHROOT_DIR}/${PP}"
 
@@ -202,6 +206,15 @@ dpkg_undo_mounts() {
     sudo rmdir ${BUILDROOT}
 }
 
+do_prepare_build_append() {
+    # Make a local copy of isar-apt repo that is not affected by other parallel builds
+    mkdir -p ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
+    rm -rf ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/*
+    cp -Rl ${REPO_ISAR_DIR} ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
+}
+
+do_prepare_build[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
+
 # Placeholder for actual dpkg_runbuild() implementation
 dpkg_runbuild() {
     die "This should never be called, overwrite it in your derived class"
@@ -224,14 +237,11 @@ def isar_export_build_settings(d):
     os.environ['DEB_BUILD_PROFILES'] = isar_deb_build_profiles(d)
 
 python do_dpkg_build() {
-    lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
-                             shared=True)
-    bb.build.exec_func("dpkg_do_mounts", d)
+    bb.build.exec_func('schroot_create_configs', d)
     try:
         bb.build.exec_func("dpkg_runbuild", d)
     finally:
-        bb.build.exec_func("dpkg_undo_mounts", d)
-        bb.utils.unlockfile(lock)
+        bb.build.exec_func('schroot_delete_configs', d)
 }
 
 addtask dpkg_build
@@ -261,7 +271,7 @@ python do_dpkg_build_setscene() {
 addtask dpkg_build_setscene
 do_dpkg_build_setscene[dirs] += "${S}/.."
 
-KEEP_INSTALLED_ON_CLEAN ?= "0"
+do_dpkg_build[depends] = "${SCHROOT_DEP}"
 
 CLEANFUNCS += "deb_clean"
 
@@ -271,15 +281,6 @@ deb_clean() {
         for d in ${DEBS}; do
             repo_del_package "${REPO_ISAR_DIR}"/"${DISTRO}" \
                 "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${d}"
-            if [ "${KEEP_INSTALLED_ON_CLEAN}" = "1" ]; then
-                continue;
-            fi
-            package=$(basename "${d}")
-            package_remove="/usr/bin/apt-get remove -y ${package%%_*}"
-            sudo -E chroot ${BUILDCHROOT_DIR} ${package_remove} || true
-            if [ "${BUILDCHROOT_DIR}" != "${BUILDCHROOT_TARGET_DIR}" ]; then
-                    sudo -E chroot ${BUILDCHROOT_TARGET_DIR} ${package_remove} || true
-            fi
         done
     fi
 }
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index 2b0b818..68172b3 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -11,6 +11,8 @@ PATCHTOOL ?= "git"
 
 GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
 
+SCHROOT_MOUNTS = "${WORKDIR}:${PP} ${GITDIR}:/home/.git-downloads"
+
 dpkg_runbuild_prepend() {
     sh -c "
         cd ${WORKDIR}/${PPS}
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 320102b..848ffad 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -39,6 +39,12 @@ dpkg_runbuild() {
     export DEB_BUILD_OPTIONS="${@ isar_deb_build_options(d)}"
     export DEB_BUILD_PROFILES="${@ isar_deb_build_profiles(d)}"
     export PARALLEL_MAKE="${PARALLEL_MAKE}"
-    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
-         /isar/build.sh ${PP}/${PPS} ${PACKAGE_ARCH}
+
+    export SBUILD_CONFIG="${SBUILD_CONFIG}"
+
+    sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
+        --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \
+        --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
+        --debbuildopts="--source-option=-I" \
+        --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
 }
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index b9fc6a0..a3783bc 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -29,6 +29,8 @@ SBUILD_CHROOT ?= "${DEBDISTRONAME}-${SCHROOT_USER}-${@os.getpid()}"
 SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}"
 SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}"
 
+SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
+
 schroot_create_configs() {
     sudo -s <<'EOSUDO'
         set -e
@@ -73,7 +75,6 @@ EOSUDO
 }
 
 sbuild_export() {
-    SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
     VAR=${1}; shift
     VAR_LINE="'${VAR}' => '${@}',"
     if [ -s "${SBUILD_CONFIG}" ]; then
@@ -83,7 +84,6 @@ sbuild_export() {
         echo "${VAR_LINE}" >> ${SBUILD_CONFIG}
         echo "};" >> ${SBUILD_CONFIG}
     fi
-    export SBUILD_CONFIG="${SBUILD_CONFIG}"
 }
 
 insert_mounts() {
-- 
2.17.1

[PATCH v10 06/19] sbuild: Support of DEB_BUILD_PROFILES

[Anton Mikanovich]

From: Uladzimir Bely <ubely@ilbers.de>

In Isar packages may use their own profiles. For example, linux-custom.inc
introduces 'nolibcdev' profile that should be passed to dpkg-buildpackage
in both cross and native builds.

By default, sbuild for cross-building uses passes "cross,nocheck" options
to dpkg-buildpackage and ignores DEB_BUILD_PROFILES environment variable.

This change makes sbuild use custom profiles even in cross-build mode.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 848ffad..1f795c7 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -37,13 +37,17 @@ dpkg_runbuild() {
     E="${@ isar_export_proxies(d)}"
     E="${@ isar_export_ccache(d)}"
     export DEB_BUILD_OPTIONS="${@ isar_deb_build_options(d)}"
-    export DEB_BUILD_PROFILES="${@ isar_deb_build_profiles(d)}"
     export PARALLEL_MAKE="${PARALLEL_MAKE}"
 
+    profiles="${@ isar_deb_build_profiles(d)}"
+    if [ ! -z "$profiles" ]; then
+        profiles=$(echo --profiles="$profiles" | sed -e 's/ \+/,/g')
+    fi
+
     export SBUILD_CONFIG="${SBUILD_CONFIG}"
 
     sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
-        --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \
+        --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
         --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
         --debbuildopts="--source-option=-I" \
         --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
-- 
2.17.1

[PATCH v10 07/19] sbuild: Support of shell exports from dpkg_runbuild_prepend

[Anton Mikanovich]

From: Uladzimir Bely <ubely@ilbers.de>

Many of recipes often use shell exports done in dpkg_run_prepend.

While sbuild is performed in isolated environment, we need a way to pass
these exported variables to it. This is done by storing environment just
before dpkg_runbuild (after prepare_build) and finding just before
the actual build what has been changed or added.

This patch allows using shell exports, but warning will be produced.
It's recommended to modify the code (in downstreams) and replace shell
exports by other methods, like using templates.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 1f795c7..b61d182 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -32,6 +32,12 @@ do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
 
 addtask devshell after do_install_builddeps
 
+DPKG_PREBUILD_ENV_FILE="${WORKDIR}/dpkg_prebuild.env"
+
+do_prepare_build_append() {
+    env > ${DPKG_PREBUILD_ENV_FILE}
+}
+
 # Build package from sources using build script
 dpkg_runbuild() {
     E="${@ isar_export_proxies(d)}"
@@ -39,6 +45,26 @@ dpkg_runbuild() {
     export DEB_BUILD_OPTIONS="${@ isar_deb_build_options(d)}"
     export PARALLEL_MAKE="${PARALLEL_MAKE}"
 
+    env | while read -r line; do
+        # Filter the same lines
+        grep -q "^${line}" ${DPKG_PREBUILD_ENV_FILE} && continue
+        # Filter some standard variables
+        echo ${line} | grep -q "^HOME=" && continue
+        echo ${line} | grep -q "^PWD=" && continue
+
+        var=$(echo "${line}" | cut -d '=' -f1)
+        value=$(echo "${line}" | cut -d '=' -f2-)
+        sbuild_export $var "$value"
+
+        # Don't warn some variables
+        [ "${var}" = "PARALLEL_MAKE" ] && continue
+        [ "${var}" = "CCACHE_DIR" ] && continue
+        [ "${var}" = "PATH_PREPEND" ] && continue
+        [ "${var}" = "DEB_BUILD_OPTIONS" ] && continue
+
+        bbwarn "Export of '${line}' detected, please migrate to templates"
+    done
+
     profiles="${@ isar_deb_build_profiles(d)}"
     if [ ! -z "$profiles" ]; then
         profiles=$(echo --profiles="$profiles" | sed -e 's/ \+/,/g')
-- 
2.17.1

[PATCH v10 08/19] dpkg: Remove builddeps install task

[Anton Mikanovich]

From: Uladzimir Bely <ubely@ilbers.de>

While builddeps are now handled interlnally by sbuild, we don't need
a task to preinstall them in buildchroot.

But for the second local (base-apt) build we need a way to keep
all dependencies in form of .deb files in DL_DIR. This is done
by executing additional commands in sbuild that copy them to/from
an externally mounted folder.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/dpkg.bbclass   | 47 ++++++++++++++++---------------------
 meta/classes/sbuild.bbclass |  1 -
 2 files changed, 20 insertions(+), 28 deletions(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index b61d182..3c7b670 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -5,33 +5,6 @@ inherit dpkg-base
 
 PACKAGE_ARCH ?= "${DISTRO_ARCH}"
 
-# Install build dependencies for package
-do_install_builddeps() {
-    dpkg_do_mounts
-    E="${@ isar_export_proxies(d)}"
-    export DEB_BUILD_OPTIONS="${@ isar_deb_build_options(d)}"
-    export DEB_BUILD_PROFILES="${@ isar_deb_build_profiles(d)}"
-    distro="${DISTRO}"
-    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
-        distro="${HOST_DISTRO}"
-    fi
-
-    deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
-        ${PP}/${PPS} ${PACKAGE_ARCH} --download-only
-    deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
-        ${PP}/${PPS} ${PACKAGE_ARCH}
-    dpkg_undo_mounts
-}
-
-addtask install_builddeps after do_prepare_build before do_dpkg_build
-do_install_builddeps[depends] += "isar-apt:do_cache_config"
-# apt and reprepro may not run in parallel, acquire the Isar lock
-do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
-
-addtask devshell after do_install_builddeps
-
 DPKG_PREBUILD_ENV_FILE="${WORKDIR}/dpkg_prebuild.env"
 
 do_prepare_build_append() {
@@ -65,6 +38,16 @@ dpkg_runbuild() {
         bbwarn "Export of '${line}' detected, please migrate to templates"
     done
 
+    distro="${DISTRO}"
+    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+        distro="${HOST_DISTRO}"
+    fi
+
+    deb_dl_dir_import "${WORKDIR}/rootfs" "${distro}"
+
+    deb_dir="/var/cache/apt/archives"
+    ext_deb_dir="${PP}/rootfs/${deb_dir}"
+
     profiles="${@ isar_deb_build_profiles(d)}"
     if [ ! -z "$profiles" ]; then
         profiles=$(echo --profiles="$profiles" | sed -e 's/ \+/,/g')
@@ -72,9 +55,19 @@ dpkg_runbuild() {
 
     export SBUILD_CONFIG="${SBUILD_CONFIG}"
 
+    echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
+
     sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
         --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
         --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
+        --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
+        --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
+        --finished-build-commands="cp -n --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
         --debbuildopts="--source-option=-I" \
         --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
+
+    deb_dl_dir_export "${WORKDIR}/rootfs" "${distro}"
+
+    # Cleanup apt artifacts
+    sudo rm -rf ${WORKDIR}/rootfs
 }
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index a3783bc..178ba1b 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -8,7 +8,6 @@ SCHROOT_MOUNTS ?= ""
 python __anonymous() {
     import pwd
     d.setVar('SCHROOT_USER', pwd.getpwuid(os.geteuid()).pw_name)
-    d.setVar('SCHROOT_USER_HOME', pwd.getpwuid(os.geteuid()).pw_dir)
 
     mode = d.getVar('ISAR_CROSS_COMPILE', True)
     distro_arch = d.getVar('DISTRO_ARCH')
-- 
2.17.1

[PATCH v10 09/19] sbuild: Add ccache support

[Anton Mikanovich]

From: Uladzimir Bely <ubely@ilbers.de>

This adds ccache support for custom packages in the same manner
as it was done previously for buildchroot-based build, by using
USE_CCACHE variable in local.conf or per-recipe.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass                     |  4 ++++
 meta/classes/sbuild.bbclass                   | 23 +++++++++++++++++++
 .../sbuild-chroot/sbuild-chroot.inc           |  1 +
 3 files changed, 28 insertions(+)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 3c7b670..d92135c 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -48,6 +48,10 @@ dpkg_runbuild() {
     deb_dir="/var/cache/apt/archives"
     ext_deb_dir="${PP}/rootfs/${deb_dir}"
 
+    if [ ${USE_CCACHE} -eq 1 ]; then
+        schroot_configure_ccache
+    fi
+
     profiles="${@ isar_deb_build_profiles(d)}"
     if [ ! -z "$profiles" ]; then
         profiles=$(echo --profiles="$profiles" | sed -e 's/ \+/,/g')
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 178ba1b..6ff41dd 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -105,3 +105,26 @@ remove_mounts() {
         done
 EOSUDO
 }
+
+schroot_configure_ccache() {
+    sudo -s <<'EOSUDO'
+        set -e
+
+        sbuild_fstab="${SBUILD_CONF_DIR}/fstab"
+
+        install --group=sbuild --mode=2775 -d ${CCACHE_DIR}
+        fstab_ccachedir="${CCACHE_DIR} /ccache none rw,bind 0 0"
+        grep -qxF "${fstab_ccachedir}" ${sbuild_fstab} || echo "${fstab_ccachedir}" >> ${sbuild_fstab}
+
+        (flock 9
+        [ -w ${CCACHE_DIR}/sbuild-setup ] || cat << END > ${CCACHE_DIR}/sbuild-setup
+#!/bin/sh
+export PATH="\$PATH_PREPEND:\$PATH"
+exec "\$@"
+END
+        chmod a+rx ${CCACHE_DIR}/sbuild-setup
+        ) 9>"${CCACHE_DIR}/sbuild-setup.lock"
+
+        echo "command-prefix=/ccache/sbuild-setup" >> "${SCHROOT_CONF_FILE}"
+EOSUDO
+}
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
index 177a8a6..74811e2 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -14,6 +14,7 @@ SBUILD_CHROOT_PREINSTALL_COMMON = " \
     fakeroot \
     build-essential \
     debhelper \
+    ccache \
 "
 
 SBUILD_CHROOT_PREINSTALL ?= " \
-- 
2.17.1

[PATCH v10 10/19] dpkg-base: Switch devshell to use schroot

[Anton Mikanovich]

From: Uladzimir Bely <ubely@ilbers.de>

While packages are now build in schroot with sbuild, devshell
task should make terminal open inside sbuild environment.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-base.bbclass                | 23 ++++++++++++++-----
 meta/classes/sbuild.bbclass                   |  3 +++
 .../sbuild-chroot/sbuild-chroot.inc           |  2 ++
 3 files changed, 22 insertions(+), 6 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index c60052d..df42e41 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -298,19 +298,30 @@ do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
 do_deploy_deb[dirs] = "${S}"
 
 python do_devshell() {
-    bb.build.exec_func('dpkg_do_mounts', d)
+    bb.build.exec_func('schroot_create_configs', d)
 
     isar_export_proxies(d)
     isar_export_ccache(d)
     isar_export_build_settings(d)
 
-    buildchroot = d.getVar('BUILDCHROOT_DIR')
+    schroot = d.getVar('SBUILD_CHROOT')
+    isar_apt = d.getVar('ISAR_APT_REPO')
     pp_pps = os.path.join(d.getVar('PP'), d.getVar('PPS'))
-    # the PATH variable is not forwarded by sudo -E.
-    termcmd = "sudo -E chroot {0} sh -c 'cd {1}; export PATH=$PATH_PREPEND:$PATH; $SHELL -i'"
-    oe_terminal(termcmd.format(buildchroot, pp_pps), "Isar devshell", d)
 
-    bb.build.exec_func('dpkg_undo_mounts', d)
+    install_deps = ":" if d.getVar('BB_CURRENTTASK') == "devshell_nodeps" else "mk-build-deps -i -t \
+        \"apt-get -y -q -o Debug::pkgProblemResolver=yes --no-install-recommends --allow-downgrades\" \
+        debian/control"
+
+    termcmd = "schroot -d / -c {0} -u root -- sh -c ' \
+        cd {1}; \
+        echo {2} > /etc/apt/sources.list.d/isar_apt.list; \
+        apt-get -y -q update; \
+        {3}; \
+        $SHELL -i \
+    '"
+    oe_terminal(termcmd.format(schroot, pp_pps, isar_apt, install_deps), "Isar devshell", d)
+
+    bb.build.exec_func('schroot_delete_configs', d)
 }
 
 addtask devshell after do_prepare_build
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 6ff41dd..2d52307 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -56,6 +56,9 @@ EOF
         fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind 0 0"
         grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}
 
+        fstab_pkgdir="${WORKDIR} /home/builder/${PN} none rw,bind 0 0"
+        grep -qxF "${fstab_pkgdir}" ${sbuild_fstab} || echo "${fstab_pkgdir}" >> ${sbuild_fstab}
+
         if [ -d ${DL_DIR} ]; then
             fstab_downloads="${DL_DIR} /downloads none rw,bind 0 0"
             grep -qxF "${fstab_downloads}" ${sbuild_fstab} || echo "${fstab_downloads}" >> ${sbuild_fstab}
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
index 74811e2..4688da1 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -15,6 +15,8 @@ SBUILD_CHROOT_PREINSTALL_COMMON = " \
     build-essential \
     debhelper \
     ccache \
+    devscripts \
+    equivs \
 "
 
 SBUILD_CHROOT_PREINSTALL ?= " \
-- 
2.17.1

[PATCH v10 11/19] dpkg-base: Switch apt_fetch and apt_unpack to use schroot

[Anton Mikanovich]

From: Uladzimir Bely <ubely@ilbers.de>

This moves downloading and unpacking deb-src to schroot environment.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 25 +++++++++----------------
 1 file changed, 9 insertions(+), 16 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index df42e41..01676f5 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -105,19 +105,12 @@ python() {
 }
 
 do_apt_fetch() {
-    dpkg_do_mounts
-    E="${@ isar_export_proxies(d)}"
-    sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update \
-        -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
-        -o Dir::Etc::SourceParts="-" \
-        -o APT::Get::List-Cleanup="0"
-
+    schroot_create_configs
     for uri in "${SRC_APT}"; do
-        sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+        schroot -d / -c ${SBUILD_CHROOT} -- \
             sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only-source source "$2"' my_script "${DISTRO}" "${uri}"
     done
-
-    dpkg_undo_mounts
+    schroot_delete_configs
 }
 
 addtask apt_fetch after do_unpack before do_apt_unpack
@@ -126,13 +119,14 @@ do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
 # Add dependency from the correct buildchroot: host or target
 do_apt_fetch[depends] = "${BUILDCHROOT_DEP}"
 
+# Add dependency from the correct schroot: host or target
+do_apt_fetch[depends] += "${SCHROOT_DEP}"
+
 do_apt_unpack() {
     rm -rf ${S}
-    dpkg_do_mounts
-    E="${@ isar_export_proxies(d)}"
-
+    schroot_create_configs
     for uri in "${SRC_APT}"; do
-        sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+        schroot -d / -c ${SBUILD_CHROOT} -- \
             sh -c ' \
                 set -e
                 dscfile="$(apt-get -y -qq --print-uris --only-source source "${2}" | cut -d " " -f2 | grep -E "*.dsc")"
@@ -141,8 +135,7 @@ do_apt_unpack() {
                 dpkg-source -x "${dscfile}" "${PPS}"' \
                     my_script "${DISTRO}" "${uri}"
     done
-
-    dpkg_undo_mounts
+    schroot_delete_configs
 }
 
 addtask apt_unpack after do_apt_fetch before do_patch
-- 
2.17.1

[PATCH v10 12/19] doc: Add sbuild-related documentation

[Anton Mikanovich]

From: Uladzimir Bely <ubely@ilbers.de>

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 doc/user_manual.md | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index bcff8d9..ee68f8b 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -86,6 +86,24 @@ apt install \
   pristine-tar
 ```
 
+Additional setup is required since `sbuild` is now used for package build.
+Install the following packages:
+```
+apt install \
+  sbuild \
+  schroot
+```
+Also, user who runs isar should be added to `sbuild` group.
+
+**NOTE:** sbuild version (<=0.78.1) packaged in Debian Buster doesn't support
+`$apt_keep_downloaded_packages` option which is required in Isar for
+populating `${DL_DIR}/deb`. So, host `sbuild` in this case should be manually
+upgraded to >=0.81.2 version from Debian Bullseye.
+
+```
+sudo gpasswd -a <username> sbuild
+```
+
 If your host is >= buster, also install the following package.
 ```
 apt install python3-distutils
-- 
2.17.1

[PATCH v10 13/19] sbuild: Use .dsc file instead of source directory

[Anton Mikanovich]

From: Uladzimir Bely <ubely@ilbers.de>

This makes build more Debian-like and avoids additional execution of
"dh clean" by sbuild before dependencies resolved and build started.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index d92135c..564e654 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -61,6 +61,10 @@ dpkg_runbuild() {
 
     echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
 
+    # Create a .dsc file from source directory to use it with sbuild
+    sh -c "cd ${WORKDIR}; dpkg-source -q -b ${PPS}"
+    DSC=$(head -n1 ${WORKDIR}/${PPS}/debian/changelog | awk '{gsub(/[()]/,""); printf "%s_%s.dsc", $1, $2}')
+
     sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
         --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
         --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
@@ -68,7 +72,7 @@ dpkg_runbuild() {
         --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
         --finished-build-commands="cp -n --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
         --debbuildopts="--source-option=-I" \
-        --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
+        --build-dir=${WORKDIR} --dist="isar" ${WORKDIR}/${DSC}
 
     deb_dl_dir_export "${WORKDIR}/rootfs" "${distro}"
 
-- 
2.17.1

[PATCH v10 14/19] sbuild: Fixed proxy support

[Anton Mikanovich]

From: Uladzimir Bely <ubely@ilbers.de>

This makes sbuild able to work via proxy.

Proxy settings passed via 'sbuild_export' seem to work only for build
stage in sbuild. Previous stages (like schroot setup, downloading deps)
don't use this values.

To work via proxy in this stages there should be environment filter
for sbuild set via sbuild.conf.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-base.bbclass |  1 +
 meta/classes/dpkg.bbclass      | 14 ++++++++++++++
 meta/classes/sbuild.bbclass    | 31 ++++++++++++++++++++++++-------
 3 files changed, 39 insertions(+), 7 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 01676f5..e879bf9 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -105,6 +105,7 @@ python() {
 }
 
 do_apt_fetch() {
+    E="${@ isar_export_proxies(d)}"
     schroot_create_configs
     for uri in "${SRC_APT}"; do
         schroot -d / -c ${SBUILD_CHROOT} -- \
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 564e654..d8c4dd8 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -35,6 +35,15 @@ dpkg_runbuild() {
         [ "${var}" = "PATH_PREPEND" ] && continue
         [ "${var}" = "DEB_BUILD_OPTIONS" ] && continue
 
+        [ "${var}" = "http_proxy" ] && continue
+        [ "${var}" = "HTTP_PROXY" ] && continue
+        [ "${var}" = "https_proxy" ] && continue
+        [ "${var}" = "HTTPS_PROXY" ] && continue
+        [ "${var}" = "ftp_proxy" ] && continue
+        [ "${var}" = "FTP_PROXY" ] && continue
+        [ "${var}" = "no_proxy" ] && continue
+        [ "${var}" = "NO_PROXY" ] && continue
+
         bbwarn "Export of '${line}' detected, please migrate to templates"
     done
 
@@ -59,6 +68,11 @@ dpkg_runbuild() {
 
     export SBUILD_CONFIG="${SBUILD_CONFIG}"
 
+    for envvar in http_proxy HTTP_PROXY https_proxy HTTPS_PROXY \
+        ftp_proxy FTP_PROXY no_proxy NO_PROXY; do
+        sbuild_add_env_filter "$envvar"
+    done
+
     echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
 
     # Create a .dsc file from source directory to use it with sbuild
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 2d52307..7d24c47 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -76,16 +76,33 @@ schroot_delete_configs() {
 EOSUDO
 }
 
+sbuild_add_env_filter() {
+    [ -w ${SBUILD_CONFIG} ] || touch ${SBUILD_CONFIG}
+
+    if ! grep -q "^\$environment_filter =" ${SBUILD_CONFIG}; then
+        echo "\$environment_filter = [" >> ${SBUILD_CONFIG}
+        echo "];" >> ${SBUILD_CONFIG}
+    fi
+
+    FILTER=${1}
+
+    sed -i -e "/'\^${FILTER}\\$/d" \
+        -e "/^\$environment_filter =.*/a '^${FILTER}\$'," ${SBUILD_CONFIG}
+}
+
 sbuild_export() {
-    VAR=${1}; shift
-    VAR_LINE="'${VAR}' => '${@}',"
-    if [ -s "${SBUILD_CONFIG}" ]; then
-        sed -i -e "\$i\\" -e "${VAR_LINE}" ${SBUILD_CONFIG}
-    else
-        echo "\$build_environment = {" > ${SBUILD_CONFIG}
-        echo "${VAR_LINE}" >> ${SBUILD_CONFIG}
+    [ -w ${SBUILD_CONFIG} ] || touch ${SBUILD_CONFIG}
+
+    if ! grep -q "^\$build_environment =" ${SBUILD_CONFIG}; then
+        echo "\$build_environment = {" >> ${SBUILD_CONFIG}
         echo "};" >> ${SBUILD_CONFIG}
     fi
+
+    VAR=${1}; shift
+    VAR_LINE="'${VAR}' => '${@}',"
+
+    sed -i -e "/^'${VAR}' =>/d" ${SBUILD_CONFIG} \
+        -e "/^\$build_environment =.*/a ${VAR_LINE}" ${SBUILD_CONFIG}
 }
 
 insert_mounts() {
-- 
2.17.1

[PATCH v10 15/19] sbuild: Fix debsrc_download for packages dependencies

[Anton Mikanovich]

From: Uladzimir Bely <ubely@ilbers.de>

With sbuild, build deps are internally installed in temporary sbuild
environment,so they don't appear in any external dpkg.log files.

With "cache-deb-src" base repo feature enabled, these such packages
are considered not to belong to current build, so their src packages
are not downloaded at image postprocess stage.

The patch fixes this by extracting dpkg.log files from the sbuild
environment to one common log file. This consolidated log is then
used at rootfs_postprocess to get source packages.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/deb-dl-dir.bbclass | 4 ++++
 meta/classes/dpkg.bbclass       | 6 +++++-
 meta/classes/sbuild.bbclass     | 9 +++++++++
 3 files changed, 18 insertions(+), 1 deletion(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index eace510..b74018f 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -14,6 +14,10 @@ is_not_part_of_current_build() {
     # build using the below method.
     local output="$( grep -hs "status installed ${package}:${arch} ${version}$" \
             "${IMAGE_ROOTFS}"/var/log/dpkg.log \
+            "${SCHROOT_HOST_DIR}"/var/log/dpkg.log \
+            "${SCHROOT_TARGET_DIR}"/var/log/dpkg.log \
+            "${SCHROOT_HOST_DIR}"/tmp/dpkg_common.log \
+            "${SCHROOT_TARGET_DIR}"/tmp/dpkg_common.log \
             "${BUILDCHROOT_HOST_DIR}"/var/log/dpkg.log \
             "${BUILDCHROOT_TARGET_DIR}"/var/log/dpkg.log | head -1 )"
 
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index d8c4dd8..a0544d5 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -55,7 +55,8 @@ dpkg_runbuild() {
     deb_dl_dir_import "${WORKDIR}/rootfs" "${distro}"
 
     deb_dir="/var/cache/apt/archives"
-    ext_deb_dir="${PP}/rootfs/${deb_dir}"
+    ext_root="${PP}/rootfs"
+    ext_deb_dir="${ext_root}${deb_dir}"
 
     if [ ${USE_CCACHE} -eq 1 ]; then
         schroot_configure_ccache
@@ -82,12 +83,15 @@ dpkg_runbuild() {
     sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
         --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
         --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
+        --chroot-setup-commands="rm -f /var/log/dpkg.log" \
         --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
         --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
         --finished-build-commands="cp -n --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
+        --finished-build-commands="cp /var/log/dpkg.log ${ext_root}/dpkg_partial.log" \
         --debbuildopts="--source-option=-I" \
         --build-dir=${WORKDIR} --dist="isar" ${WORKDIR}/${DSC}
 
+    sbuild_dpkg_log_export "${WORKDIR}/rootfs/dpkg_partial.log"
     deb_dl_dir_export "${WORKDIR}/rootfs" "${distro}"
 
     # Cleanup apt artifacts
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 7d24c47..a78b15b 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -148,3 +148,12 @@ END
         echo "command-prefix=/ccache/sbuild-setup" >> "${SCHROOT_CONF_FILE}"
 EOSUDO
 }
+
+sbuild_dpkg_log_export() {
+    export dpkg_partial_log="${1}"
+
+    ( flock 9
+    set -e
+    cat ${dpkg_partial_log} >> ${SCHROOT_DIR}/tmp/dpkg_common.log
+    )  9>"${SCHROOT_DIR}/tmp/dpkg_common.log.lock"
+}
-- 
2.17.1

[PATCH v10 16/19] fix: support build of packages with epoch version

[Anton Mikanovich]

From: Felix Moessbauer <felix.moessbauer@siemens.com>

This patch reworks the logic how the DSC file for building
with sbuild is located.
By that, we do not rely on implementation details of dpkg-source
regarding the name of the generated .dsc file.
Instead, we remove all <package>*.dsc files in advance, let dpkg-source
generate the file and use a find to locate the generated one.

By that, package versions with epochs (e.g. "1:1.0.0") are
supported.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/dpkg.bbclass | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index a0544d5..3dda7d2 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -77,8 +77,10 @@ dpkg_runbuild() {
     echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
 
     # Create a .dsc file from source directory to use it with sbuild
+    DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
+    find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -delete
     sh -c "cd ${WORKDIR}; dpkg-source -q -b ${PPS}"
-    DSC=$(head -n1 ${WORKDIR}/${PPS}/debian/changelog | awk '{gsub(/[()]/,""); printf "%s_%s.dsc", $1, $2}')
+    DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -print)
 
     sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
         --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
@@ -89,7 +91,7 @@ dpkg_runbuild() {
         --finished-build-commands="cp -n --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
         --finished-build-commands="cp /var/log/dpkg.log ${ext_root}/dpkg_partial.log" \
         --debbuildopts="--source-option=-I" \
-        --build-dir=${WORKDIR} --dist="isar" ${WORKDIR}/${DSC}
+        --build-dir=${WORKDIR} --dist="isar" ${DSC_FILE}
 
     sbuild_dpkg_log_export "${WORKDIR}/rootfs/dpkg_partial.log"
     deb_dl_dir_export "${WORKDIR}/rootfs" "${distro}"
-- 
2.17.1

[PATCH v10 17/19] always create apt-cache dirs in deb_dl_dir_import

[Anton Mikanovich]

From: Felix Moessbauer <felix.moessbauer@siemens.com>

This patch ensures that the deb_dl_dir_import function
always creates the apt-cache directories under ${PP}/rootfs,
no matter if files are in the download cache or not.

By that, downstream tasks can safely assume that the apt-cache
is created and place files there.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/deb-dl-dir.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index b74018f..83014d9 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -78,8 +78,8 @@ debsrc_download() {
 deb_dl_dir_import() {
     export pc="${DEBDIR}/${2}"
     export rootfs="${1}"
-    [ ! -d "${pc}" ] && return 0
     sudo mkdir -p "${rootfs}"/var/cache/apt/archives/
+    [ ! -d "${pc}" ] && return 0
     flock -s "${pc}".lock -c '
         set -e
         printenv | grep -q BB_VERBOSE_LOGS && set -x
-- 
2.17.1

[PATCH v10 18/19] avoid absolute SCHROOT_* paths to improve caching

[Anton Mikanovich]

From: Felix Moessbauer <felix.moessbauer@siemens.com>

This patch reworks the logic to setup the SCHROOT_* bitbake
variables by making sure nothing is expanded.
By that, no absolute paths into the build host are exposed,
which finally makes the recipe cachable.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/sbuild.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index a78b15b..f9bfaaf 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -14,11 +14,11 @@ python __anonymous() {
     if mode == "0" or d.getVar('HOST_ARCH') ==  distro_arch or \
        (d.getVar('HOST_DISTRO') == "debian-stretch" and distro_arch == "i386"):
         d.setVar('SBUILD_HOST_ARCH', distro_arch)
-        d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_TARGET_DIR'))
+        d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_TARGET_DIR', False))
         dep = "sbuild-chroot-target:do_build"
     else:
         d.setVar('SBUILD_HOST_ARCH', d.getVar('HOST_ARCH'))
-        d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_HOST_DIR'))
+        d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_HOST_DIR', False))
         dep = "sbuild-chroot-host:do_build"
     d.setVar('SCHROOT_DEP', dep)
 }
-- 
2.17.1

[PATCH v10 19/19] dpkg-base: Cleanup on schroot fail

[Anton Mikanovich]

Cleanup schroot configs in case schroot exit with non zero state.
Handle all possible interrupt cases with trap.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index e879bf9..c543eb2 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -107,6 +107,13 @@ python() {
 do_apt_fetch() {
     E="${@ isar_export_proxies(d)}"
     schroot_create_configs
+
+    schroot_cleanup() {
+        schroot_delete_configs
+    }
+    trap 'exit 1' INT HUP QUIT TERM ALRM USR1
+    trap 'schroot_cleanup' EXIT
+
     for uri in "${SRC_APT}"; do
         schroot -d / -c ${SBUILD_CHROOT} -- \
             sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only-source source "$2"' my_script "${DISTRO}" "${uri}"
@@ -126,6 +133,13 @@ do_apt_fetch[depends] += "${SCHROOT_DEP}"
 do_apt_unpack() {
     rm -rf ${S}
     schroot_create_configs
+
+    schroot_cleanup() {
+        schroot_delete_configs
+    }
+    trap 'exit 1' INT HUP QUIT TERM ALRM USR1
+    trap 'schroot_cleanup' EXIT
+
     for uri in "${SRC_APT}"; do
         schroot -d / -c ${SBUILD_CHROOT} -- \
             sh -c ' \
-- 
2.17.1

Re: [PATCH v10 00/19] Sbuild/Schroot migration

[Anton Mikanovich]

05.05.2022 17:39, Anton Mikanovich wrote:
> This is a patchset showing how sbuild/schroot tools can be integrated
> into Isar build system.
>
> Base schroot images are created with sbuild-chroot-target or, in case
> of cross-build, sbuild-chroot-host recipes. These images are based on
> isar-bootstrap ones, but include some build-related stuff preinstalled.
>
> To use these changes you need to have sbuild and schroot installed and
> configured. User who runs build should be added to sbuild group.
>
> Currently, sbuild-related stuff is supported in latest kas.
>
> If Gilab CI is used, the following configuration changes are required:
> - external dir for schroot overlay should be set in the config
> /etc/gitlab-runner/config.toml:
>
> volumes = ["/path/to/overlay:/var/lib/schroot/union/overlay"]
>
> The patchset was tested with 'meta-iot2050' and 'xenomai-images'
> downstreams in combination with kas image mentioned above.
>
> Current limitations:
> - parallel building of several packages requires more free space in
> comparison with buildchroot-based architecture. This happens due
> the sbuild architecture which uses some basic layer (common for all
> packages) and per-package separate layer (where builddeps are downloaded
> and installed).

Does anything speak against merging this patchset?
If not, we would like to do it next week.

RE: [PATCH v10 00/19] Sbuild/Schroot migration

[Moessbauer, Felix]

> -----Original Message-----
> From: Anton Mikanovich <amikan@ilbers.de>
> Sent: Thursday, May 12, 2022 2:55 PM
> To: isar-users@googlegroups.com
> Cc: Schmidt, Adriaan (T CED SES-DE) <adriaan.schmidt@siemens.com>;
> Moessbauer, Felix (T CED SES-DE) <felix.moessbauer@siemens.com>; Kiszka,
> Jan (T CED) <jan.kiszka@siemens.com>; Schild, Henning (T CED SES-DE)
> <henning.schild@siemens.com>; Baurzhan Ismagulov <ibr@ilbers.de>
> Subject: Re: [PATCH v10 00/19] Sbuild/Schroot migration
> 
> 05.05.2022 17:39, Anton Mikanovich wrote:
> > This is a patchset showing how sbuild/schroot tools can be integrated
> > into Isar build system.
> >
> > Base schroot images are created with sbuild-chroot-target or, in case
> > of cross-build, sbuild-chroot-host recipes. These images are based on
> > isar-bootstrap ones, but include some build-related stuff preinstalled.
> >
> > To use these changes you need to have sbuild and schroot installed and
> > configured. User who runs build should be added to sbuild group.
> >
> > Currently, sbuild-related stuff is supported in latest kas.
> >
> > If Gilab CI is used, the following configuration changes are required:
> > - external dir for schroot overlay should be set in the config
> > /etc/gitlab-runner/config.toml:
> >
> > volumes = ["/path/to/overlay:/var/lib/schroot/union/overlay"]
> >
> > The patchset was tested with 'meta-iot2050' and 'xenomai-images'
> > downstreams in combination with kas image mentioned above.
> >
> > Current limitations:
> > - parallel building of several packages requires more free space in
> > comparison with buildchroot-based architecture. This happens due the
> > sbuild architecture which uses some basic layer (common for all
> > packages) and per-package separate layer (where builddeps are
> > downloaded and installed).
> 
> Does anything speak against merging this patchset?
> If not, we would like to do it next week.

Hi,

thanks for all the effort regarding the sbuild support.
We tested that on a couple of layers and it mostly worked quite well.
The following aspects are worth thinking about, but should not result in another delay in the integration of sbuild:

Removal of do_install_builddeps task: Please add a bold statement about that in the API changelog

Manual actions in a package schroot: Some layers (like meta-coral) require additional work to be done inside the buildchroot to prepare things prior to the debuild / sbuild.
Mounting the correct schroot and working there works, but this is totally undocumented ground. We should really guide the ISAR users how to do that correctly (otherwise a ton of partially working patterns will evolve downstream).
A public example, where this is necessary can be found in [1] (still pre-sbuild)

Setup and teardown of schroots:
The teardown should always be executed, no matter if the job was successful or not.
Currently this is handled by shell-traps, but we should really think about simplifying that for the users.
Either a post-task handler could be registered for cleanup, or we use bitbake functions which are called from python with a try / catch / finally pattern.
Maybe @Adriaan can comment on that, as he has built up quite some experience around cleanups when working on the mount patch series.

Integration of other patch series:
At least personally I would really prefer to see other patch series being merged first.
Examples are: cachability improvements, sstate scripts, mounting of /dev/pts (at least the bugfix series).
It is likely that pre-sbuild ISAR will be around for quite some time as layers need time to adopt.

New ISAR release: Once Sbuilder is in, the old ISAR should be tagged and the in-flight version number of ISAR should be increased.

Happy Coding!
Felix

[1] https://github.com/siemens/meta-coral/blob/main/classes/dpkg-bazel.bbclass

Re: [PATCH v10 00/19] Sbuild/Schroot migration

[Baurzhan Ismagulov]

On Thu, May 12, 2022 at 03:55:19PM +0000, Moessbauer, Felix wrote:
> thanks for all the effort regarding the sbuild support.
> We tested that on a couple of layers and it mostly worked quite well.
> The following aspects are worth thinking about, but should not result in another delay in the integration of sbuild:

Thanks for the input, I also think we should find a good way to merge sbuild
soon.


> Removal of do_install_builddeps task: Please add a bold statement about that in the API changelog

The patch has been sent.


> Manual actions in a package schroot: Some layers (like meta-coral) require additional work to be done inside the buildchroot to prepare things prior to the debuild / sbuild.
> Mounting the correct schroot and working there works, but this is totally undocumented ground. We should really guide the ISAR users how to do that correctly (otherwise a ton of partially working patterns will evolve downstream).
> A public example, where this is necessary can be found in [1] (still pre-sbuild)

In short, we should avoid any buildchroot preparations. The Debian way is to
specify anything that is needed in the build dependencies, so that a dumb
package builder can build any package in a cleanroom environment without
knowing what is inside. We've done this for meta-iot2050.

This works well for Debian which doesn't have many flavors (except a couple of
kernels for x86 and arm). In the layers, we often want to customize stuff.
Currently I'm aware of the environment way (like DEB_BUILD_OPTIONS), but that
is handled in debian/rules; right now I don't know how to handle other stuff
like build dependencies. If you have specific examples, I think we should
discuss it with the upstream -- maybe they'd have suggestions, or we could
collaborate on improvements.

For the imagers, seems we'll have an schroot session API for now, but I'd
prefer not to extend it to sbuild if at all possible.


> Setup and teardown of schroots:
> The teardown should always be executed, no matter if the job was successful or not.
> Currently this is handled by shell-traps, but we should really think about simplifying that for the users.
> Either a post-task handler could be registered for cleanup, or we use bitbake functions which are called from python with a try / catch / finally pattern.
> Maybe @Adriaan can comment on that, as he has built up quite some experience around cleanups when working on the mount patch series.

Well, we do prefer python and try-catch. We've tried that with mount rework and
it works well. The issue is that downstreams are affected, and we couldn't find
an elegant way to migrate away from shell scripts in downstreams. Longer-term,
I think we'll have to do that, shell tasks are interfering with too much stuff.

Till then, I'm afraid we have to live with shell. And its error handling, is,
well, traps :) .


> Integration of other patch series:
> At least personally I would really prefer to see other patch series being merged first.
> Examples are: cachability improvements, sstate scripts, mounting of /dev/pts (at least the bugfix series).

Some of the stuff has been merged.


> It is likely that pre-sbuild ISAR will be around for quite some time as layers need time to adopt.
> 
> New ISAR release: Once Sbuilder is in, the old ISAR should be tagged and the in-flight version number of ISAR should be increased.

Which in-flight version number do you mean? I'm only aware of the release info
in /etc, which uses git to get the tag.


With kind regards,
Baurzhan.