[PATCH v4 00/12] Sbuild/Schroot migration

[PATCH v4 00/12] Sbuild/Schroot migration

[Uladzimir Bely]

This is a patchset showing how sbuild/schroot tools can be integrated
into Isar build system.

This patchset should be applied on top of "preparation" patchset
named "Avoid using shell environment during the build".

Base schroot image is created with sbuild-chroot-target (or
sbuild-chroot-host in case of cross-build) recipe. These images are based
on isar-bootstrap ones, but include some build-related stuff preinstalled.

To use this changes you need to have sbuild and schroot installed and
configured. Current user should be added to sbuild group.

If 'kas-container' is used, it should be modified to support sbuild:
- /var/lib/schroot/union/overlay should be externally mounted (-v option)

Currently, official `kas` images don't include all required software, so
there was an alternative docker image prepared:
`docker pull ghcr.io/wiselord/kas/kas-isar:next-sbuild`

To use it with KAS:
- export KAS_IMAGE_VERSION=3.0.0-sbuild
- export KAS_CONTAINER_IMAGE_PATH=ghcr.io/wiselord/kas

If 'gitlab' is used, the similar changes are required:
- 'image: ghcr.io/wiselord/kas/kas-isar:3.0.0-sbuild' in .gitlab-ci.yml
- external dir for schroot overlay should also be specified in
/etc/gitlab-runner/config.toml:
volumes = ["/path/to/overlay:/var/lib/schroot/union/overlay"]

The patchset was tested with 'meta-iot2050' and 'xenomai-images'
downstreams. Some patches are required for these downstreams,
appropriate pull requests will be created soon.

Current limitations:
- parallel building of several packages requires more free space in
comparison with buildchroot-based architecture. This happens due
the sbuild architecture which uses some basic layer (common for all
packages) and per-package separate layer (where builddeps are downloaded
and installed).

Changes since v3:
- dpkg_do_mounts() and dpkg_undo_mounts() are not removed for downstreams
compatibility;
- dpkg_build_export is used for adjusting sbuild environment;
- DEB_BUILD_PROFILES support for sbuild is fixed in case of cross-build.
Changes since v2:
- patches reworked/squashed for easier reading and understanding;
- fixed building foreigh architectures with kas-docker;
- implemented support of ccache;
- fixed devshell and devshell_nodeps,
Changes since v1:
- parallel builds with different BUILD_DIR are supported;
- parallel multiconfig targets in one build are supported;
- per-task schroot configuration in /etc/schroot/ is now used;
- patchset now passes Jenkins CI (so patches changes RFC => PATCH).

Anton Mikanovich (1):
  dpkg: Build packages with sbuild

Uladzimir Bely (11):
  dpkg-gbp: Use separate command to export tarball
  dpkg-gbp: Use host tools for dsc preparation
  sbuild: Add recipes for host and target rootfs to run sbuild
  sbuild: Introduce a class for another build method
  sbuild: support of DEB_BUILD_PROFILES
  sbuild: add ccache support
  dpkg: Remove builddeps install task.
  dpkg-base: Switch devshell to use schroot
  dpkg-base: Switch apt_fetch and apt_unpack to use schroot
  dpkg-base: Cleanup from buildchroot parts.
  doc: Add sbuild-related documentation

 doc/user_manual.md                            |  22 ++-
 meta/classes/dpkg-base.bbclass                |  81 ++++----
 meta/classes/dpkg-gbp.bbclass                 |  26 +--
 meta/classes/dpkg.bbclass                     |  62 +++---
 meta/classes/sbuild.bbclass                   | 185 ++++++++++++++++++
 meta/conf/bitbake.conf                        |   2 +
 .../sbuild-chroot/sbuild-chroot-host.bb       |  13 ++
 .../sbuild-chroot/sbuild-chroot-target.bb     |  10 +
 .../sbuild-chroot/sbuild-chroot.inc           |  39 ++++
 9 files changed, 359 insertions(+), 81 deletions(-)
 create mode 100644 meta/classes/sbuild.bbclass
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc

-- 
2.20.1

[PATCH v4 01/12] dpkg-gbp: Use separate command to export tarball

[Uladzimir Bely]

We don't actually build the package with gbp, but only prepare it for
the building with external dpkg-buildpackage command. In case there is
no need in real build we can perform only export which will produce
upstream tarball to pass for later building.
This allows to use any build tool for later processing.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-gbp.bbclass | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index d956e8c3..ddf23ca5 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -30,5 +30,9 @@ do_install_builddeps_append() {
 }
 
 dpkg_runbuild_prepend() {
-    export GBP_PREFIX="gbp buildpackage --git-ignore-new ${GBP_EXTRA_OPTIONS} --git-builder="
+    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+        sh -c "cd ${PP}/${PPS} && gbp buildpackage --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}"
+    # NOTE: `buildpackage --git-builder=/bin/true --git-pristine-tar` is used
+    # for compatibility with gbp version froms debian-stretch. In newer distros
+    # it's possible to use a subcommand `export-orig --pristine-tar`
 }
-- 
2.20.1

[PATCH v4 02/12] dpkg-gbp: Use host tools for dsc preparation

[Uladzimir Bely]

Instead of preinstalling gbp-related packages in sbuild chroot
we will use their host versions in order to keep sbuild environment
as clean as possible.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 doc/user_manual.md            |  4 +++-
 meta/classes/dpkg-gbp.bbclass | 24 ++++--------------------
 2 files changed, 7 insertions(+), 21 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index caaab8cc..97270167 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -79,7 +79,9 @@ apt install \
   qemu \
   qemu-user-static \
   reprepro \
-  sudo
+  sudo \
+  git-buildpackage \
+  pristine-tar
 ```
 
 If your host is >= buster, also install the following package.
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index ddf23ca5..2fc37b2f 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -9,29 +9,13 @@ S = "${WORKDIR}/git"
 
 PATCHTOOL ?= "git"
 
-GBP_DEPENDS ?= "git-buildpackage pristine-tar"
 GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
 
-do_install_builddeps_append() {
-    dpkg_do_mounts
-    distro="${DISTRO}"
-    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
-       distro="${HOST_DISTRO}"
-    fi
-    deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} \
-        apt-get install -y -o Debug::pkgProblemResolver=yes \
-                        --no-install-recommends --download-only ${GBP_DEPENDS}
-    deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} \
-        apt-get install -y -o Debug::pkgProblemResolver=yes \
-                        --no-install-recommends ${GBP_DEPENDS}
-    dpkg_undo_mounts
-}
-
 dpkg_runbuild_prepend() {
-    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
-        sh -c "cd ${PP}/${PPS} && gbp buildpackage --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}"
+    sh -c "
+        cd ${WORKDIR}/${PPS}
+        gbp buildpackage --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}
+    "
     # NOTE: `buildpackage --git-builder=/bin/true --git-pristine-tar` is used
     # for compatibility with gbp version froms debian-stretch. In newer distros
     # it's possible to use a subcommand `export-orig --pristine-tar`
-- 
2.20.1

[PATCH v4 03/12] sbuild: Add recipes for host and target rootfs to run sbuild

[Uladzimir Bely]

Similar to buildchroot, we need a separate rootfs to be used
for schroot.

It's based on bootstrapped rootfs, but includes several
common build-related packages.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 .../sbuild-chroot/sbuild-chroot-host.bb       | 13 +++++++
 .../sbuild-chroot/sbuild-chroot-target.bb     | 10 ++++++
 .../sbuild-chroot/sbuild-chroot.inc           | 36 +++++++++++++++++++
 3 files changed, 59 insertions(+)
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc

diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
new file mode 100644
index 00000000..aa82846b
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
@@ -0,0 +1,13 @@
+# Root filesystem for packages building
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2021 ilbers GmbH
+
+DESCRIPTION = "Isar sbuild/schroot filesystem for host"
+
+SBUILD_VARIANT = "host"
+
+require sbuild-chroot.inc
+
+ROOTFS_ARCH = "${HOST_ARCH}"
+ROOTFS_DISTRO = "${HOST_DISTRO}"
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
new file mode 100644
index 00000000..d75d783b
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
@@ -0,0 +1,10 @@
+# Root filesystem for packages building
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2021 ilbers GmbH
+
+DESCRIPTION = "Isar sbuild/schroot filesystem for target"
+
+SBUILD_VARIANT = "target"
+
+require sbuild-chroot.inc
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
new file mode 100644
index 00000000..177a8a6d
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -0,0 +1,36 @@
+# Common part for build chroot filesystem.
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2021 ilbers GmbH
+
+LICENSE = "gpl-2.0"
+LIC_FILES_CHKSUM = "file://${LAYERDIR_core}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
+
+PV = "1.0"
+
+inherit rootfs
+
+SBUILD_CHROOT_PREINSTALL_COMMON = " \
+    fakeroot \
+    build-essential \
+    debhelper \
+"
+
+SBUILD_CHROOT_PREINSTALL ?= " \
+    ${SBUILD_CHROOT_PREINSTALL_COMMON} \
+"
+
+SBUILD_CHROOT_DIR = "${WORKDIR}/rootfs"
+ROOTFSDIR = "${SBUILD_CHROOT_DIR}"
+ROOTFS_PACKAGES = "${SBUILD_CHROOT_PREINSTALL}"
+
+# We don't need /etc/apt/sources.list.d/isar-apt.list' while it's handled by sbuild
+ROOTFS_CONFIGURE_COMMAND_remove = "rootfs_configure_isar_apt"
+
+DEPLOY_SCHROOT = "${@d.getVar('SCHROOT_' + d.getVar('SBUILD_VARIANT').upper() + '_DIR')}"
+
+do_sbuildchroot_deploy[dirs] = "${DEPLOY_DIR}/schroot-${SBUILD_VARIANT}"
+do_sbuildchroot_deploy() {
+    ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_SCHROOT}"
+}
+addtask sbuildchroot_deploy before do_build after do_rootfs
-- 
2.20.1

[PATCH v4 04/12] sbuild: Introduce a class for another build method

[Uladzimir Bely]

This also adds mounts for base-apt inside schroot and adds import/export
deb files to/from schroot. So that it becomes possible to run second
 `cached` build from local base-apt repo.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/sbuild.bbclass | 160 ++++++++++++++++++++++++++++++++++++
 meta/conf/bitbake.conf      |   2 +
 2 files changed, 162 insertions(+)
 create mode 100644 meta/classes/sbuild.bbclass

diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
new file mode 100644
index 00000000..e4341a6f
--- /dev/null
+++ b/meta/classes/sbuild.bbclass
@@ -0,0 +1,160 @@
+# This software is a part of ISAR.
+# Copyright (C) 2021 ilbers GmbH
+
+SCHROOT_CONF ?= "/etc/schroot"
+
+SCHROOT_MOUNTS ?= ""
+
+python __anonymous() {
+    import pwd
+    d.setVar('SCHROOT_USER', pwd.getpwuid(os.geteuid()).pw_name)
+    d.setVar('SCHROOT_USER_HOME', pwd.getpwuid(os.geteuid()).pw_dir)
+
+    mode = d.getVar('ISAR_CROSS_COMPILE', True)
+    distro_arch = d.getVar('DISTRO_ARCH')
+    if mode == "0" or d.getVar('HOST_ARCH') ==  distro_arch or \
+       (d.getVar('HOST_DISTRO') == "debian-stretch" and distro_arch == "i386"):
+        d.setVar('SBUILD_HOST_ARCH', distro_arch)
+        d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_TARGET_DIR'))
+        dep = "sbuild-chroot-target:do_build"
+    else:
+        d.setVar('SBUILD_HOST_ARCH', d.getVar('HOST_ARCH'))
+        d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_HOST_DIR'))
+        dep = "sbuild-chroot-host:do_build"
+    d.setVar('SCHROOT_DEP', dep)
+}
+
+SBUILD_CHROOT ?= "${DEBDISTRONAME}-${SCHROOT_USER}-${@os.getpid()}"
+SBUILD_CHROOT_RW ?= "${SBUILD_CHROOT}-rw"
+
+SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}"
+SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}"
+SCHROOT_CONF_FILE_RW ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}-rw"
+
+schroot_create_configs() {
+    sudo -s <<'EOSUDO'
+        set -e
+
+        cat << EOF > "${SCHROOT_CONF_FILE}"
+[${SBUILD_CHROOT}]
+type=directory
+directory=${SCHROOT_DIR}
+profile=${SBUILD_CHROOT}
+users=${SCHROOT_USER}
+groups=root,sbuild
+root-users=${SCHROOT_USER}
+root-groups=root,sbuild
+source-root-users=${SCHROOT_USER}
+source-root-groups=root,sbuild
+union-type=overlay
+preserve-environment=true
+EOF
+
+        cat << EOF > "${SCHROOT_CONF_FILE_RW}"
+[${SBUILD_CHROOT_RW}]
+type=directory
+directory=${SCHROOT_DIR}
+profile=${SBUILD_CHROOT}
+users=${SCHROOT_USER}
+groups=root,sbuild
+root-users=${SCHROOT_USER}
+root-groups=root,sbuild
+preserve-environment=true
+EOF
+
+        mkdir -p "${SCHROOT_DIR}/etc/apt/preferences.d"
+        cat << EOF > "${SCHROOT_DIR}/etc/apt/preferences.d/isar-apt"
+Package: *
+Pin: release n=${DEBDISTRONAME}
+Pin-Priority: 1000
+EOF
+
+        # Prepare mount points
+        cp -rf "${SCHROOT_CONF}/sbuild" "${SBUILD_CONF_DIR}"
+        sbuild_fstab="${SBUILD_CONF_DIR}/fstab"
+
+        fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind 0 0"
+        grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}
+
+        if [ -d ${DL_DIR} ]; then
+            fstab_downloads="${DL_DIR} /downloads none rw,bind 0 0"
+            grep -qxF "${fstab_downloads}" ${sbuild_fstab} || echo "${fstab_downloads}" >> ${sbuild_fstab}
+        fi
+EOSUDO
+}
+
+schroot_delete_configs() {
+    sudo -s <<'EOSUDO'
+        set -e
+        if [ -d "${SBUILD_CONF_DIR}" ]; then
+            rm -rf "${SBUILD_CONF_DIR}"
+        fi
+        rm -f "${SCHROOT_DIR}/etc/apt/preferences.d/isar-apt"
+        rm -f "${SCHROOT_CONF_FILE}"
+        rm -f "${SCHROOT_CONF_FILE_RW}"
+EOSUDO
+}
+
+sbuild_export() {
+    SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
+    VAR=${1}; shift
+    VAR_LINE="'${VAR}' => '${@}',"
+    if [ -s "${SBUILD_CONFIG}" ]; then
+        sed -i -e "\$i\\" -e "${VAR_LINE}" ${SBUILD_CONFIG}
+    else
+        echo "\$build_environment = {" > ${SBUILD_CONFIG}
+        echo "${VAR_LINE}" >> ${SBUILD_CONFIG}
+        echo "};" >> ${SBUILD_CONFIG}
+    fi
+    export SBUILD_CONFIG="${SBUILD_CONFIG}"
+}
+
+schroot_install() {
+    schroot_create_configs
+    APTS="$1"
+
+    distro="${DISTRO}"
+    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+       distro="${HOST_DISTRO}"
+    fi
+
+    deb_dl_dir_import ${SCHROOT_DIR} ${distro}
+    schroot -d / -c ${SBUILD_CHROOT_RW} -u root -- \
+        apt install -y -o Debug::pkgProblemResolver=yes \
+                    --no-install-recommends --download-only ${APTS}
+    deb_dl_dir_export ${SCHROOT_DIR} ${distro}
+
+    schroot -d / -c ${SBUILD_CHROOT_RW} -u root -- \
+        apt install -y -o Debug::pkgProblemResolver=yes \
+                    --no-install-recommends ${APTS}
+    schroot_delete_configs
+}
+
+insert_mounts() {
+    sudo -s <<'EOSUDO'
+        set -e
+        for mp in ${SCHROOT_MOUNTS}; do
+            FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
+            grep -qxF "${FSTAB_LINE}" ${SBUILD_CONF_DIR}/fstab || \
+                echo "${FSTAB_LINE}" >> ${SBUILD_CONF_DIR}/fstab
+        done
+EOSUDO
+}
+
+remove_mounts() {
+    sudo -s <<'EOSUDO'
+        set -e
+        for mp in ${SCHROOT_MOUNTS}; do
+            FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
+            sed -i "\|${FSTAB_LINE}|d" ${SBUILD_CONF_DIR}/fstab
+        done
+EOSUDO
+}
+
+schroot_run() {
+    schroot_create_configs
+    insert_mounts
+    schroot $@
+    remove_mounts
+    schroot_delete_configs
+}
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index cd1c4a64..c1458f05 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -58,6 +58,8 @@ SSTATE_DIR ?= "${TOPDIR}/sstate-cache"
 SSTATE_MANIFESTS = "${TMPDIR}/sstate-control/${DISTRO}-${DISTRO_ARCH}"
 BUILDCHROOT_HOST_DIR = "${DEPLOY_DIR_BUILDCHROOT}-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
 BUILDCHROOT_TARGET_DIR = "${DEPLOY_DIR_BUILDCHROOT}-target/${DISTRO}-${DISTRO_ARCH}"
+SCHROOT_HOST_DIR = "${DEPLOY_DIR}/schroot-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
+SCHROOT_TARGET_DIR = "${DEPLOY_DIR}/schroot-target/${DISTRO}-${DISTRO_ARCH}"
 SDKCHROOT_DIR = "${DEPLOY_DIR_SDKCHROOT}/${DISTRO}-${DISTRO_ARCH}"
 CACHE = "${TMPDIR}/cache"
 KERNEL_FILE ?= "vmlinuz"
-- 
2.20.1

[PATCH v4 05/12] dpkg: Build packages with sbuild

[Uladzimir Bely]

From: Anton Mikanovich <amikan@ilbers.de>

Use previously exported to schroot rootfs to build the package with
sbuild tool. Provide isar-apt as extra repository to be automatically
add to apt sources.

Also added /home/.git-downloads mount while it's used
as git alternates location.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 16 +++++++++++++++-
 meta/classes/dpkg-gbp.bbclass  |  2 ++
 meta/classes/dpkg.bbclass      | 13 +++++++++++--
 3 files changed, 28 insertions(+), 3 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 76b1290a..e1c27b9e 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -4,6 +4,7 @@
 #
 # SPDX-License-Identifier: MIT
 
+inherit sbuild
 inherit buildchroot
 inherit debianize
 inherit terminal
@@ -14,6 +15,8 @@ DEPENDS ?= ""
 
 DEPENDS_append_riscv64 = "${@' crossbuild-essential-riscv64' if d.getVar('ISAR_CROSS_COMPILE', True) == '1' and d.getVar('PN') != 'crossbuild-essential-riscv64' else ''}"
 
+ISAR_APT_REPO ?= "deb [trusted=yes] file:///home/builder/${PN}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO} ${DEBDISTRONAME} main"
+
 python do_adjust_git() {
     import subprocess
 
@@ -196,13 +199,22 @@ dpkg_undo_mounts() {
     sudo rmdir ${BUILDROOT}
 }
 
+do_prepare_build_append() {
+    # Make a local copy of isar-apt repo that is not affected by other parallel builds
+    mkdir -p ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
+    rm -rf ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/*
+    cp -Rl ${REPO_ISAR_DIR} ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
+}
+
+do_prepare_build[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
+
 # Basic export implementation supporting multiword values
 dpkg_build_export() {
     var=$1
     shift
     value=$@
 
-    export $var="$value"
+    sbuild_export $var "$value"
 }
 
 # Placeholder for actual dpkg_runbuild() implementation
@@ -250,6 +262,8 @@ python do_dpkg_build_setscene() {
 addtask dpkg_build_setscene
 do_dpkg_build_setscene[dirs] += "${S}/.."
 
+do_dpkg_build[depends] = "${SCHROOT_DEP}"
+
 KEEP_INSTALLED_ON_CLEAN ?= "0"
 
 CLEANFUNCS += "deb_clean"
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index 2fc37b2f..7eda9b03 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -11,6 +11,8 @@ PATCHTOOL ?= "git"
 
 GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
 
+SCHROOT_MOUNTS = "${WORKDIR}:${PP} ${GITDIR}:/home/.git-downloads"
+
 dpkg_runbuild_prepend() {
     sh -c "
         cd ${WORKDIR}/${PPS}
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index e8c7e2eb..66539f48 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -29,11 +29,20 @@ do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
 
 addtask devshell after do_install_builddeps
 
+
 # Build package from sources using build script
 dpkg_runbuild() {
     E="${@ isar_export_proxies(d)}"
     E="${@ isar_export_ccache(d)}"
     dpkg_build_export PARALLEL_MAKE "${PARALLEL_MAKE}"
-    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
-         /isar/build.sh ${PP}/${PPS} ${PACKAGE_ARCH}
+
+    schroot_create_configs
+
+    sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
+        --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \
+        --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
+        --debbuildopts="--source-option=-I" \
+        --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
+
+    schroot_delete_configs
 }
-- 
2.20.1

[PATCH v4 06/12] sbuild: support of DEB_BUILD_PROFILES

[Uladzimir Bely]

In Isar packages may use their own profiles. For example, linux-custom.inc
introduces 'nolibcdev' profile that should be passed to dpkg-buildpackage
in both cross and native builds.

By default, sbuild for cross-building uses passes "cross,nocheck" options
to dpkg-buildpackage and ignores DEB_BUILD_PROFILES environment variable.

This change makes sbuild use custom profiles even in cross-build mode.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 66539f48..8647d077 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -38,8 +38,16 @@ dpkg_runbuild() {
 
     schroot_create_configs
 
+    profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 | cut -d "'" -f 4)
+    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+        profiles="${profiles} cross nocheck"
+    fi
+    if [ ! -z "$profiles" ]; then
+        profiles=$(echo --profiles="$profiles" | sed -e 's/ \+/,/g')
+    fi
+
     sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
-        --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \
+        --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
         --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
         --debbuildopts="--source-option=-I" \
         --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
-- 
2.20.1

[PATCH v4 07/12] sbuild: add ccache support

[Uladzimir Bely]

This adds ccache support for custom packages in the same manner
as it was done previously for buildchroot-based build, by using
USE_CCACHE variable in local.conf or per-recipe.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass                     |  4 ++++
 meta/classes/sbuild.bbclass                   | 22 +++++++++++++++++++
 .../sbuild-chroot/sbuild-chroot.inc           |  1 +
 3 files changed, 27 insertions(+)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 8647d077..e761078d 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -38,6 +38,10 @@ dpkg_runbuild() {
 
     schroot_create_configs
 
+    if [ ${USE_CCACHE} -eq 1 ]; then
+        schroot_configure_ccache
+    fi
+
     profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 | cut -d "'" -f 4)
     if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
         profiles="${profiles} cross nocheck"
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index e4341a6f..c9676bdf 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -158,3 +158,25 @@ schroot_run() {
     remove_mounts
     schroot_delete_configs
 }
+
+schroot_configure_ccache() {
+    sudo -s <<'EOSUDO'
+        set -e
+
+        sbuild_fstab="${SBUILD_CONF_DIR}/fstab"
+
+        install --group=sbuild --mode=2775 -d ${CCACHE_DIR}
+        fstab_ccachedir="${CCACHE_DIR} /ccache none rw,bind 0 0"
+        grep -qxF "${fstab_ccachedir}" ${sbuild_fstab} || echo "${fstab_ccachedir}" >> ${sbuild_fstab}
+
+        cat << END > ${CCACHE_DIR}/sbuild-setup
+#!/bin/sh
+export CCACHE_DIR=/ccache
+export PATH="/usr/lib/ccache:\$PATH"
+exec "\$@"
+END
+        chmod a+rx ${CCACHE_DIR}/sbuild-setup
+
+        echo "command-prefix=/ccache/sbuild-setup" >> "${SCHROOT_CONF_FILE}"
+EOSUDO
+}
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
index 177a8a6d..74811e22 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -14,6 +14,7 @@ SBUILD_CHROOT_PREINSTALL_COMMON = " \
     fakeroot \
     build-essential \
     debhelper \
+    ccache \
 "
 
 SBUILD_CHROOT_PREINSTALL ?= " \
-- 
2.20.1

[PATCH v4 08/12] dpkg: Remove builddeps install task.

[Uladzimir Bely]

While builddeps are now handled interlnally by sbuild, we don't need
a task to preinstall them in buildchroot.

But for the second local (base-apt) build we need a way to keep
all dependencies in form of .deb files in DL_DIR. This is done
by executing additional commands in sbuild that copy them to/from
an externally mounted folder.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass | 43 ++++++++++++++++++---------------------
 1 file changed, 20 insertions(+), 23 deletions(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index e761078d..38845a16 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -5,30 +5,7 @@ inherit dpkg-base
 
 PACKAGE_ARCH ?= "${DISTRO_ARCH}"
 
-# Install build dependencies for package
-do_install_builddeps() {
-    dpkg_do_mounts
-    E="${@ isar_export_proxies(d)}"
-    distro="${DISTRO}"
-    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
-       distro="${HOST_DISTRO}"
-    fi
-    deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
-        ${PP}/${PPS} ${PACKAGE_ARCH} --download-only
-    deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
-        ${PP}/${PPS} ${PACKAGE_ARCH}
-    dpkg_undo_mounts
-}
-
-addtask install_builddeps after do_prepare_build before do_dpkg_build
 do_install_builddeps[depends] += "isar-apt:do_cache_config"
-# apt and reprepro may not run in parallel, acquire the Isar lock
-do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
-
-addtask devshell after do_install_builddeps
-
 
 # Build package from sources using build script
 dpkg_runbuild() {
@@ -38,6 +15,21 @@ dpkg_runbuild() {
 
     schroot_create_configs
 
+    distro="${DISTRO}"
+    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+        distro="${HOST_DISTRO}"
+    fi
+
+    deb_dl_dir_import "${WORKDIR}/rootfs" "${distro}"
+
+    deb_dir="/var/cache/apt/archives/"
+    ext_deb_dir="/home/builder/${PN}/rootfs/${deb_dir}"
+
+    ( flock 9
+        grep -qxF '$apt_keep_downloaded_packages = 1;' ${SCHROOT_USER_HOME}/.sbuildrc ||
+            echo '$apt_keep_downloaded_packages = 1;' >> ${SCHROOT_USER_HOME}/.sbuildrc
+    ) 9>"${TMPDIR}/sbuildrc.lock"
+
     if [ ${USE_CCACHE} -eq 1 ]; then
         schroot_configure_ccache
     fi
@@ -53,8 +45,13 @@ dpkg_runbuild() {
     sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
         --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
         --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
+        --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
+        --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
+        --finished-build-commands="cp -n --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
         --debbuildopts="--source-option=-I" \
         --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
 
+    deb_dl_dir_export "${WORKDIR}/rootfs" "${distro}"
+
     schroot_delete_configs
 }
-- 
2.20.1

[PATCH v4 09/12] dpkg-base: Switch devshell to use schroot

[Uladzimir Bely]

While packages are now build in schroot with sbuild, devshell
task should make terminal open inside sbuild environment.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-base.bbclass                | 23 ++++++++++++++-----
 meta/classes/sbuild.bbclass                   |  3 +++
 .../sbuild-chroot/sbuild-chroot.inc           |  2 ++
 3 files changed, 22 insertions(+), 6 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index e1c27b9e..4f8c3256 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -300,18 +300,29 @@ do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
 do_deploy_deb[dirs] = "${S}"
 
 python do_devshell() {
-    bb.build.exec_func('dpkg_do_mounts', d)
+    bb.build.exec_func('schroot_create_configs', d)
 
     isar_export_proxies(d)
     isar_export_ccache(d)
 
-    buildchroot = d.getVar('BUILDCHROOT_DIR')
+    schroot = d.getVar('SBUILD_CHROOT')
+    isar_apt = d.getVar('ISAR_APT_REPO')
     pp_pps = os.path.join(d.getVar('PP'), d.getVar('PPS'))
-    # the PATH variable is not forwarded by sudo -E.
-    termcmd = "sudo -E chroot {0} sh -c 'cd {1}; export PATH=$PATH_PREPEND:$PATH; $SHELL -i'"
-    oe_terminal(termcmd.format(buildchroot, pp_pps), "Isar devshell", d)
 
-    bb.build.exec_func('dpkg_undo_mounts', d)
+    install_deps = ":" if d.getVar('BB_CURRENTTASK') == "devshell_nodeps" else "mk-build-deps -i -t \
+        \"apt-get -y -q -o Debug::pkgProblemResolver=yes --no-install-recommends --allow-downgrades\" \
+        debian/control"
+
+    termcmd = "schroot -d / -c {0} -u root -- sh -c ' \
+        cd {1}; \
+        echo {2} > /etc/apt/sources.list.d/isar_apt.list; \
+        apt-get -y -q update; \
+        {3}; \
+        $SHELL -i \
+    '"
+    oe_terminal(termcmd.format(schroot, pp_pps, isar_apt, install_deps), "Isar devshell", d)
+
+    bb.build.exec_func('schroot_delete_configs', d)
 }
 
 addtask devshell after do_prepare_build
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index c9676bdf..b4024852 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -76,6 +76,9 @@ EOF
         fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind 0 0"
         grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}
 
+        fstab_pkgdir="${WORKDIR} /home/builder/${PN} none rw,bind 0 0"
+        grep -qxF "${fstab_pkgdir}" ${sbuild_fstab} || echo "${fstab_pkgdir}" >> ${sbuild_fstab}
+
         if [ -d ${DL_DIR} ]; then
             fstab_downloads="${DL_DIR} /downloads none rw,bind 0 0"
             grep -qxF "${fstab_downloads}" ${sbuild_fstab} || echo "${fstab_downloads}" >> ${sbuild_fstab}
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
index 74811e22..4688da1d 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -15,6 +15,8 @@ SBUILD_CHROOT_PREINSTALL_COMMON = " \
     build-essential \
     debhelper \
     ccache \
+    devscripts \
+    equivs \
 "
 
 SBUILD_CHROOT_PREINSTALL ?= " \
-- 
2.20.1

[PATCH v4 10/12] dpkg-base: Switch apt_fetch and apt_unpack to use schroot

[Uladzimir Bely]

This moves downloading and unpacking deb-src to schroot environment.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 25 +++++++++----------------
 1 file changed, 9 insertions(+), 16 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 4f8c3256..ea914cd4 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -99,19 +99,12 @@ python() {
 }
 
 do_apt_fetch() {
-    dpkg_do_mounts
-    E="${@ isar_export_proxies(d)}"
-    sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update \
-        -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
-        -o Dir::Etc::SourceParts="-" \
-        -o APT::Get::List-Cleanup="0"
-
+    schroot_create_configs
     for uri in "${SRC_APT}"; do
-        sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+        schroot -d / -c ${SBUILD_CHROOT} -- \
             sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only-source source "$2"' my_script "${DISTRO}" "${uri}"
     done
-
-    dpkg_undo_mounts
+    schroot_delete_configs
 }
 
 addtask apt_fetch after do_unpack before do_apt_unpack
@@ -120,13 +113,14 @@ do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
 # Add dependency from the correct buildchroot: host or target
 do_apt_fetch[depends] = "${BUILDCHROOT_DEP}"
 
+# Add dependency from the correct schroot: host or target
+do_apt_fetch[depends] += "${SCHROOT_DEP}"
+
 do_apt_unpack() {
     rm -rf ${S}
-    dpkg_do_mounts
-    E="${@ isar_export_proxies(d)}"
-
+    schroot_create_configs
     for uri in "${SRC_APT}"; do
-        sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+        schroot -d / -c ${SBUILD_CHROOT} -- \
             sh -c ' \
                 set -e
                 dscfile="$(apt-get -y -qq --print-uris --only-source source "${2}" | cut -d " " -f2 | grep -E "*.dsc")"
@@ -135,8 +129,7 @@ do_apt_unpack() {
                 dpkg-source -x "${dscfile}" "${PPS}"' \
                     my_script "${DISTRO}" "${uri}"
     done
-
-    dpkg_undo_mounts
+    schroot_delete_configs
 }
 
 addtask apt_unpack after do_apt_fetch before do_patch
-- 
2.20.1

[PATCH v4 11/12] dpkg-base: Cleanup from buildchroot parts.

[Uladzimir Bely]

This removes remaining buildchroot-related tasks and variables
from dpkg-base class while there are not used anymore.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 19 +++----------------
 meta/classes/dpkg.bbclass      |  4 ----
 2 files changed, 3 insertions(+), 20 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index ea914cd4..513d5217 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -167,6 +167,7 @@ addtask prepare_build after do_patch do_transform_template before do_dpkg_build
 # If Isar recipes depend on each other, they typically need the package
 # deployed to isar-apt
 do_prepare_build[deptask] = "do_deploy_deb"
+do_prepare_build[depends] = "${SCHROOT_DEP}"
 
 BUILDROOT = "${BUILDCHROOT_DIR}/${PP}"
 
@@ -216,14 +217,11 @@ dpkg_runbuild() {
 }
 
 python do_dpkg_build() {
-    lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
-                             shared=True)
-    bb.build.exec_func("dpkg_do_mounts", d)
+    bb.build.exec_func('schroot_create_configs', d)
     try:
         bb.build.exec_func("dpkg_runbuild", d)
     finally:
-        bb.build.exec_func("dpkg_undo_mounts", d)
-        bb.utils.unlockfile(lock)
+        bb.build.exec_func('schroot_delete_configs', d)
 }
 
 addtask dpkg_build
@@ -257,8 +255,6 @@ do_dpkg_build_setscene[dirs] += "${S}/.."
 
 do_dpkg_build[depends] = "${SCHROOT_DEP}"
 
-KEEP_INSTALLED_ON_CLEAN ?= "0"
-
 CLEANFUNCS += "deb_clean"
 
 deb_clean() {
@@ -267,15 +263,6 @@ deb_clean() {
         for d in ${DEBS}; do
             repo_del_package "${REPO_ISAR_DIR}"/"${DISTRO}" \
                 "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${d}"
-            if [ "${KEEP_INSTALLED_ON_CLEAN}" = "1" ]; then
-                continue;
-            fi
-            package=$(basename "${d}")
-            package_remove="/usr/bin/apt-get remove -y ${package%%_*}"
-            sudo -E chroot ${BUILDCHROOT_DIR} ${package_remove} || true
-            if [ "${BUILDCHROOT_DIR}" != "${BUILDCHROOT_TARGET_DIR}" ]; then
-                    sudo -E chroot ${BUILDCHROOT_TARGET_DIR} ${package_remove} || true
-            fi
         done
     fi
 }
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 38845a16..99d1a074 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -13,8 +13,6 @@ dpkg_runbuild() {
     E="${@ isar_export_ccache(d)}"
     dpkg_build_export PARALLEL_MAKE "${PARALLEL_MAKE}"
 
-    schroot_create_configs
-
     distro="${DISTRO}"
     if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
         distro="${HOST_DISTRO}"
@@ -52,6 +50,4 @@ dpkg_runbuild() {
         --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
 
     deb_dl_dir_export "${WORKDIR}/rootfs" "${distro}"
-
-    schroot_delete_configs
 }
-- 
2.20.1

[PATCH v4 12/12] doc: Add sbuild-related documentation

[Uladzimir Bely]

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 doc/user_manual.md | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 97270167..17758d17 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -84,6 +84,24 @@ apt install \
   pristine-tar
 ```
 
+Additional setup is required since `sbuild` is now used for package build.
+Install the following packages:
+```
+apt install \
+  sbuild \
+  schroot
+```
+Also, user who runs isar should be added to `sbuild` group.
+
+**NOTE:** sbuild version (<=0.78.1) packaged in Debian Buster doesn't support
+`$apt_keep_downloaded_packages` option which is required in Isar for
+populating `${DL_DIR}/deb`. So, host `sbuild` in this case should be manually
+upgraded to >=0.81.2 version from Debian Bullseye.
+
+```
+sudo gpasswd -a <username> sbuild
+```
+
 If your host is >= buster, also install the following package.
 ```
 apt install python3-distutils
-- 
2.20.1

Re: [PATCH v4 00/12] Sbuild/Schroot migration

[quirin.gylstorff]

Hi,


On 1/25/22 13:39, Uladzimir Bely wrote:
> This is a patchset showing how sbuild/schroot tools can be integrated
> into Isar build system.
> 
> This patchset should be applied on top of "preparation" patchset
> named "Avoid using shell environment during the build".
> 
> Base schroot image is created with sbuild-chroot-target (or
> sbuild-chroot-host in case of cross-build) recipe. These images are based
> on isar-bootstrap ones, but include some build-related stuff preinstalled.
> 
> To use this changes you need to have sbuild and schroot installed and
> configured. Current user should be added to sbuild group.
> 
> If 'kas-container' is used, it should be modified to support sbuild:
> - /var/lib/schroot/union/overlay should be externally mounted (-v option)
> 
> Currently, official `kas` images don't include all required software, so
> there was an alternative docker image prepared:
> `docker pull ghcr.io/wiselord/kas/kas-isar:next-sbuild`
> 
> To use it with KAS:
> - export KAS_IMAGE_VERSION=3.0.0-sbuild
> - export KAS_CONTAINER_IMAGE_PATH=ghcr.io/wiselord/kas
> 
> If 'gitlab' is used, the similar changes are required:
> - 'image: ghcr.io/wiselord/kas/kas-isar:3.0.0-sbuild' in .gitlab-ci.yml
> - external dir for schroot overlay should also be specified in
> /etc/gitlab-runner/config.toml:
> volumes = ["/path/to/overlay:/var/lib/schroot/union/overlay"]
> 
> The patchset was tested with 'meta-iot2050' and 'xenomai-images'
> downstreams. Some patches are required for these downstreams,
> appropriate pull requests will be created soon.
> 
> Current limitations:
> - parallel building of several packages requires more free space in
> comparison with buildchroot-based architecture. This happens due
> the sbuild architecture which uses some basic layer (common for all
> packages) and per-package separate layer (where builddeps are downloaded
> and installed).
> 
> Changes since v3:
> - dpkg_do_mounts() and dpkg_undo_mounts() are not removed for downstreams
> compatibility;
> - dpkg_build_export is used for adjusting sbuild environment;
> - DEB_BUILD_PROFILES support for sbuild is fixed in case of cross-build.
> Changes since v2:
> - patches reworked/squashed for easier reading and understanding;
> - fixed building foreigh architectures with kas-docker;
> - implemented support of ccache;
> - fixed devshell and devshell_nodeps,
> Changes since v1:
> - parallel builds with different BUILD_DIR are supported;
> - parallel multiconfig targets in one build are supported;
> - per-task schroot configuration in /etc/schroot/ is now used;
> - patchset now passes Jenkins CI (so patches changes RFC => PATCH).
> 
> Anton Mikanovich (1):
>    dpkg: Build packages with sbuild
> 
> Uladzimir Bely (11):
>    dpkg-gbp: Use separate command to export tarball
>    dpkg-gbp: Use host tools for dsc preparation
>    sbuild: Add recipes for host and target rootfs to run sbuild
>    sbuild: Introduce a class for another build method
>    sbuild: support of DEB_BUILD_PROFILES
>    sbuild: add ccache support
>    dpkg: Remove builddeps install task.
>    dpkg-base: Switch devshell to use schroot
>    dpkg-base: Switch apt_fetch and apt_unpack to use schroot
>    dpkg-base: Cleanup from buildchroot parts.
>    doc: Add sbuild-related documentation
> 
>   doc/user_manual.md                            |  22 ++-
>   meta/classes/dpkg-base.bbclass                |  81 ++++----
>   meta/classes/dpkg-gbp.bbclass                 |  26 +--
>   meta/classes/dpkg.bbclass                     |  62 +++---
>   meta/classes/sbuild.bbclass                   | 185 ++++++++++++++++++
>   meta/conf/bitbake.conf                        |   2 +
>   .../sbuild-chroot/sbuild-chroot-host.bb       |  13 ++
>   .../sbuild-chroot/sbuild-chroot-target.bb     |  10 +
>   .../sbuild-chroot/sbuild-chroot.inc           |  39 ++++
>   9 files changed, 359 insertions(+), 81 deletions(-)
>   create mode 100644 meta/classes/sbuild.bbclass
>   create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
>   create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
>   create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
> 

any thoughts on the clean source behavior[1].

"When running sbuild from within an unpacked source tree, run the 
'clean' target before generating the source package. This might require 
some of the build dependencies necessary for running the 'clean' target 
to be installed on the host machine. Only disable if you start from a 
clean checkout and you know what you are doing."[1]
[1]: 
https://manpages.debian.org/unstable/sbuild/sbuild.conf.5.en.html#CLEAN_SOURCE

If this is necessary you need to install dh-*(e.g. dh-lua, dh-python) in 
the build environment.

Quirin

Re: [PATCH v4 00/12] Sbuild/Schroot migration

[Jan Kiszka]

On 04.02.22 14:39, quirin.gylstorff@siemens.com wrote:
> Hi,
> 
> 
> On 1/25/22 13:39, Uladzimir Bely wrote:
>> This is a patchset showing how sbuild/schroot tools can be integrated
>> into Isar build system.
>>
>> This patchset should be applied on top of "preparation" patchset
>> named "Avoid using shell environment during the build".
>>
>> Base schroot image is created with sbuild-chroot-target (or
>> sbuild-chroot-host in case of cross-build) recipe. These images are based
>> on isar-bootstrap ones, but include some build-related stuff
>> preinstalled.
>>
>> To use this changes you need to have sbuild and schroot installed and
>> configured. Current user should be added to sbuild group.
>>
>> If 'kas-container' is used, it should be modified to support sbuild:
>> - /var/lib/schroot/union/overlay should be externally mounted (-v option)
>>
>> Currently, official `kas` images don't include all required software, so
>> there was an alternative docker image prepared:
>> `docker pull ghcr.io/wiselord/kas/kas-isar:next-sbuild`
>>
>> To use it with KAS:
>> - export KAS_IMAGE_VERSION=3.0.0-sbuild
>> - export KAS_CONTAINER_IMAGE_PATH=ghcr.io/wiselord/kas
>>
>> If 'gitlab' is used, the similar changes are required:
>> - 'image: ghcr.io/wiselord/kas/kas-isar:3.0.0-sbuild' in .gitlab-ci.yml
>> - external dir for schroot overlay should also be specified in
>> /etc/gitlab-runner/config.toml:
>> volumes = ["/path/to/overlay:/var/lib/schroot/union/overlay"]
>>
>> The patchset was tested with 'meta-iot2050' and 'xenomai-images'
>> downstreams. Some patches are required for these downstreams,
>> appropriate pull requests will be created soon.
>>
>> Current limitations:
>> - parallel building of several packages requires more free space in
>> comparison with buildchroot-based architecture. This happens due
>> the sbuild architecture which uses some basic layer (common for all
>> packages) and per-package separate layer (where builddeps are downloaded
>> and installed).
>>
>> Changes since v3:
>> - dpkg_do_mounts() and dpkg_undo_mounts() are not removed for downstreams
>> compatibility;
>> - dpkg_build_export is used for adjusting sbuild environment;
>> - DEB_BUILD_PROFILES support for sbuild is fixed in case of cross-build.
>> Changes since v2:
>> - patches reworked/squashed for easier reading and understanding;
>> - fixed building foreigh architectures with kas-docker;
>> - implemented support of ccache;
>> - fixed devshell and devshell_nodeps,
>> Changes since v1:
>> - parallel builds with different BUILD_DIR are supported;
>> - parallel multiconfig targets in one build are supported;
>> - per-task schroot configuration in /etc/schroot/ is now used;
>> - patchset now passes Jenkins CI (so patches changes RFC => PATCH).
>>
>> Anton Mikanovich (1):
>>    dpkg: Build packages with sbuild
>>
>> Uladzimir Bely (11):
>>    dpkg-gbp: Use separate command to export tarball
>>    dpkg-gbp: Use host tools for dsc preparation
>>    sbuild: Add recipes for host and target rootfs to run sbuild
>>    sbuild: Introduce a class for another build method
>>    sbuild: support of DEB_BUILD_PROFILES
>>    sbuild: add ccache support
>>    dpkg: Remove builddeps install task.
>>    dpkg-base: Switch devshell to use schroot
>>    dpkg-base: Switch apt_fetch and apt_unpack to use schroot
>>    dpkg-base: Cleanup from buildchroot parts.
>>    doc: Add sbuild-related documentation
>>
>>   doc/user_manual.md                            |  22 ++-
>>   meta/classes/dpkg-base.bbclass                |  81 ++++----
>>   meta/classes/dpkg-gbp.bbclass                 |  26 +--
>>   meta/classes/dpkg.bbclass                     |  62 +++---
>>   meta/classes/sbuild.bbclass                   | 185 ++++++++++++++++++
>>   meta/conf/bitbake.conf                        |   2 +
>>   .../sbuild-chroot/sbuild-chroot-host.bb       |  13 ++
>>   .../sbuild-chroot/sbuild-chroot-target.bb     |  10 +
>>   .../sbuild-chroot/sbuild-chroot.inc           |  39 ++++
>>   9 files changed, 359 insertions(+), 81 deletions(-)
>>   create mode 100644 meta/classes/sbuild.bbclass
>>   create mode 100644
>> meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
>>   create mode 100644
>> meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
>>   create mode 100644
>> meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
>>
> 
> any thoughts on the clean source behavior[1].
> 
> "When running sbuild from within an unpacked source tree, run the
> 'clean' target before generating the source package. This might require
> some of the build dependencies necessary for running the 'clean' target
> to be installed on the host machine. Only disable if you start from a
> clean checkout and you know what you are doing."[1]
> [1]:
> https://manpages.debian.org/unstable/sbuild/sbuild.conf.5.en.html#CLEAN_SOURCE
> 
> 
> If this is necessary you need to install dh-*(e.g. dh-lua, dh-python) in
> the build environment.
> 

Then we would need to expand the host side (builder) dependency list.
And also kas-isar.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux