[PATCH v7 00/13] Sbuild/Schroot migration

[PATCH v7 00/13] Sbuild/Schroot migration

[Uladzimir Bely]

This is a patchset showing how sbuild/schroot tools can be integrated
into Isar build system.

This patchset should be applied on top of "preparation" patchset
named "Avoid using shell environment during the build".

Base schroot image is created with sbuild-chroot-target (or
sbuild-chroot-host in case of cross-build) recipe. These images are based
on isar-bootstrap ones, but include some build-related stuff preinstalled.

To use this changes you need to have sbuild and schroot installed and
configured. Current user should be added to sbuild group.

If 'kas-container' is used, it should be modified to support sbuild:
- /var/lib/schroot/union/overlay should be externally mounted (-v option)

Currently, sbuild-related stuff is supported in kas-isar:next image

To use the patchset with KAS:
- export KAS_IMAGE_VERSION=next

If 'gitlab' is used, the similar changes are required:
- `next` tag for kas image should be set
- external dir for schroot overlay should also be specified in
/etc/gitlab-runner/config.toml:
volumes = ["/path/to/overlay:/var/lib/schroot/union/overlay"]

The patchset was tested with 'meta-iot2050' and 'xenomai-images'
downstreams in combination with kas image mentioned above.

Current limitations:
- parallel building of several packages requires more free space in
comparison with buildchroot-based architecture. This happens due
the sbuild architecture which uses some basic layer (common for all
packages) and per-package separate layer (where builddeps are downloaded
and installed).

Changes since v6:
- added patch that switches sbuild from using source directory to using
.dsc file. The file is generated before sbuild call by `dpkg-source`.

Changes since v5:
- warning about shell exports in dpkg_runbuild_prepend. This should
avoided be reworked to templates.

Changes since v4:
- consider shell exports done in dpkg_runbuild_prepend and pass them
to sbuild environment;
- fixed ccache work.

Changes since v3:
- dpkg_do_mounts() and dpkg_undo_mounts() are not removed for downstreams
compatibility;
- dpkg_build_export is used for adjusting sbuild environment;
- DEB_BUILD_PROFILES support for sbuild is fixed in case of cross-build.

Changes since v2:
- patches reworked/squashed for easier reading and understanding;
- fixed building foreigh architectures with kas-docker;
- implemented support of ccache;
- fixed devshell and devshell_nodeps.

Changes since v1:
- parallel builds with different BUILD_DIR are supported;
- parallel multiconfig targets in one build are supported;
- per-task schroot configuration in /etc/schroot/ is now used;
- patchset now passes Jenkins CI (so patches changes RFC => PATCH).

Anton Mikanovich (1):
  dpkg: Build packages with sbuild

Uladzimir Bely (12):
  dpkg-gbp: Use separate command to export tarball
  dpkg-gbp: Use host tools for dsc preparation
  sbuild: Add recipes for host and target rootfs to run sbuild
  sbuild: Introduce a class for another build method
  sbuild: support of DEB_BUILD_PROFILES
  sbuild: support of shell exports from dpkg_runbuild_prepend
  dpkg: Remove builddeps install task.
  sbuild: add ccache support
  dpkg-base: Switch devshell to use schroot
  dpkg-base: Switch apt_fetch and apt_unpack to use schroot
  doc: Add sbuild-related documentation
  sbuild: Use .dsc file instead of source directory.

 doc/user_manual.md                            |  22 ++-
 meta/classes/dpkg-base.bbclass                |  79 ++++----
 meta/classes/dpkg-gbp.bbclass                 |  26 +--
 meta/classes/dpkg.bbclass                     |  92 ++++++---
 meta/classes/sbuild.bbclass                   | 184 ++++++++++++++++++
 meta/conf/bitbake.conf                        |   2 +
 .../sbuild-chroot/sbuild-chroot-host.bb       |  13 ++
 .../sbuild-chroot/sbuild-chroot-target.bb     |  10 +
 .../sbuild-chroot/sbuild-chroot.inc           |  39 ++++
 9 files changed, 386 insertions(+), 81 deletions(-)
 create mode 100644 meta/classes/sbuild.bbclass
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc

-- 
2.20.1

[PATCH v7 01/13] dpkg-gbp: Use separate command to export tarball

[Uladzimir Bely]

We don't actually build the package with gbp, but only prepare it for
the building with external dpkg-buildpackage command. In case there is
no need in real build we can perform only export which will produce
upstream tarball to pass for later building.
This allows to use any build tool for later processing.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-gbp.bbclass | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index d956e8c3..ddf23ca5 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -30,5 +30,9 @@ do_install_builddeps_append() {
 }
 
 dpkg_runbuild_prepend() {
-    export GBP_PREFIX="gbp buildpackage --git-ignore-new ${GBP_EXTRA_OPTIONS} --git-builder="
+    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+        sh -c "cd ${PP}/${PPS} && gbp buildpackage --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}"
+    # NOTE: `buildpackage --git-builder=/bin/true --git-pristine-tar` is used
+    # for compatibility with gbp version froms debian-stretch. In newer distros
+    # it's possible to use a subcommand `export-orig --pristine-tar`
 }
-- 
2.20.1

[PATCH v7 02/13] dpkg-gbp: Use host tools for dsc preparation

[Uladzimir Bely]

Instead of preinstalling gbp-related packages in sbuild chroot
we will use their host versions in order to keep sbuild environment
as clean as possible.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 doc/user_manual.md            |  4 +++-
 meta/classes/dpkg-gbp.bbclass | 24 ++++--------------------
 2 files changed, 7 insertions(+), 21 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 4d4fa763..df73ef89 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -80,7 +80,9 @@ apt install \
   qemu \
   qemu-user-static \
   reprepro \
-  sudo
+  sudo \
+  git-buildpackage \
+  pristine-tar
 ```
 
 If your host is >= buster, also install the following package.
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index ddf23ca5..2fc37b2f 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -9,29 +9,13 @@ S = "${WORKDIR}/git"
 
 PATCHTOOL ?= "git"
 
-GBP_DEPENDS ?= "git-buildpackage pristine-tar"
 GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
 
-do_install_builddeps_append() {
-    dpkg_do_mounts
-    distro="${DISTRO}"
-    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
-       distro="${HOST_DISTRO}"
-    fi
-    deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} \
-        apt-get install -y -o Debug::pkgProblemResolver=yes \
-                        --no-install-recommends --download-only ${GBP_DEPENDS}
-    deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} \
-        apt-get install -y -o Debug::pkgProblemResolver=yes \
-                        --no-install-recommends ${GBP_DEPENDS}
-    dpkg_undo_mounts
-}
-
 dpkg_runbuild_prepend() {
-    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
-        sh -c "cd ${PP}/${PPS} && gbp buildpackage --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}"
+    sh -c "
+        cd ${WORKDIR}/${PPS}
+        gbp buildpackage --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}
+    "
     # NOTE: `buildpackage --git-builder=/bin/true --git-pristine-tar` is used
     # for compatibility with gbp version froms debian-stretch. In newer distros
     # it's possible to use a subcommand `export-orig --pristine-tar`
-- 
2.20.1

[PATCH v7 03/13] sbuild: Add recipes for host and target rootfs to run sbuild

[Uladzimir Bely]

Similar to buildchroot, we need a separate rootfs to be used
for schroot.

It's based on bootstrapped rootfs, but includes several
common build-related packages.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 .../sbuild-chroot/sbuild-chroot-host.bb       | 13 +++++++
 .../sbuild-chroot/sbuild-chroot-target.bb     | 10 ++++++
 .../sbuild-chroot/sbuild-chroot.inc           | 36 +++++++++++++++++++
 3 files changed, 59 insertions(+)
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc

diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
new file mode 100644
index 00000000..aa82846b
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
@@ -0,0 +1,13 @@
+# Root filesystem for packages building
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2021 ilbers GmbH
+
+DESCRIPTION = "Isar sbuild/schroot filesystem for host"
+
+SBUILD_VARIANT = "host"
+
+require sbuild-chroot.inc
+
+ROOTFS_ARCH = "${HOST_ARCH}"
+ROOTFS_DISTRO = "${HOST_DISTRO}"
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
new file mode 100644
index 00000000..d75d783b
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
@@ -0,0 +1,10 @@
+# Root filesystem for packages building
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2021 ilbers GmbH
+
+DESCRIPTION = "Isar sbuild/schroot filesystem for target"
+
+SBUILD_VARIANT = "target"
+
+require sbuild-chroot.inc
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
new file mode 100644
index 00000000..177a8a6d
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -0,0 +1,36 @@
+# Common part for build chroot filesystem.
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2021 ilbers GmbH
+
+LICENSE = "gpl-2.0"
+LIC_FILES_CHKSUM = "file://${LAYERDIR_core}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
+
+PV = "1.0"
+
+inherit rootfs
+
+SBUILD_CHROOT_PREINSTALL_COMMON = " \
+    fakeroot \
+    build-essential \
+    debhelper \
+"
+
+SBUILD_CHROOT_PREINSTALL ?= " \
+    ${SBUILD_CHROOT_PREINSTALL_COMMON} \
+"
+
+SBUILD_CHROOT_DIR = "${WORKDIR}/rootfs"
+ROOTFSDIR = "${SBUILD_CHROOT_DIR}"
+ROOTFS_PACKAGES = "${SBUILD_CHROOT_PREINSTALL}"
+
+# We don't need /etc/apt/sources.list.d/isar-apt.list' while it's handled by sbuild
+ROOTFS_CONFIGURE_COMMAND_remove = "rootfs_configure_isar_apt"
+
+DEPLOY_SCHROOT = "${@d.getVar('SCHROOT_' + d.getVar('SBUILD_VARIANT').upper() + '_DIR')}"
+
+do_sbuildchroot_deploy[dirs] = "${DEPLOY_DIR}/schroot-${SBUILD_VARIANT}"
+do_sbuildchroot_deploy() {
+    ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_SCHROOT}"
+}
+addtask sbuildchroot_deploy before do_build after do_rootfs
-- 
2.20.1

[PATCH v7 04/13] sbuild: Introduce a class for another build method

[Uladzimir Bely]

This also adds mounts for base-apt inside schroot and adds import/export
deb files to/from schroot. So that it becomes possible to run second
 `cached` build from local base-apt repo.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/sbuild.bbclass | 160 ++++++++++++++++++++++++++++++++++++
 meta/conf/bitbake.conf      |   2 +
 2 files changed, 162 insertions(+)
 create mode 100644 meta/classes/sbuild.bbclass

diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
new file mode 100644
index 00000000..e4341a6f
--- /dev/null
+++ b/meta/classes/sbuild.bbclass
@@ -0,0 +1,160 @@
+# This software is a part of ISAR.
+# Copyright (C) 2021 ilbers GmbH
+
+SCHROOT_CONF ?= "/etc/schroot"
+
+SCHROOT_MOUNTS ?= ""
+
+python __anonymous() {
+    import pwd
+    d.setVar('SCHROOT_USER', pwd.getpwuid(os.geteuid()).pw_name)
+    d.setVar('SCHROOT_USER_HOME', pwd.getpwuid(os.geteuid()).pw_dir)
+
+    mode = d.getVar('ISAR_CROSS_COMPILE', True)
+    distro_arch = d.getVar('DISTRO_ARCH')
+    if mode == "0" or d.getVar('HOST_ARCH') ==  distro_arch or \
+       (d.getVar('HOST_DISTRO') == "debian-stretch" and distro_arch == "i386"):
+        d.setVar('SBUILD_HOST_ARCH', distro_arch)
+        d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_TARGET_DIR'))
+        dep = "sbuild-chroot-target:do_build"
+    else:
+        d.setVar('SBUILD_HOST_ARCH', d.getVar('HOST_ARCH'))
+        d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_HOST_DIR'))
+        dep = "sbuild-chroot-host:do_build"
+    d.setVar('SCHROOT_DEP', dep)
+}
+
+SBUILD_CHROOT ?= "${DEBDISTRONAME}-${SCHROOT_USER}-${@os.getpid()}"
+SBUILD_CHROOT_RW ?= "${SBUILD_CHROOT}-rw"
+
+SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}"
+SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}"
+SCHROOT_CONF_FILE_RW ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}-rw"
+
+schroot_create_configs() {
+    sudo -s <<'EOSUDO'
+        set -e
+
+        cat << EOF > "${SCHROOT_CONF_FILE}"
+[${SBUILD_CHROOT}]
+type=directory
+directory=${SCHROOT_DIR}
+profile=${SBUILD_CHROOT}
+users=${SCHROOT_USER}
+groups=root,sbuild
+root-users=${SCHROOT_USER}
+root-groups=root,sbuild
+source-root-users=${SCHROOT_USER}
+source-root-groups=root,sbuild
+union-type=overlay
+preserve-environment=true
+EOF
+
+        cat << EOF > "${SCHROOT_CONF_FILE_RW}"
+[${SBUILD_CHROOT_RW}]
+type=directory
+directory=${SCHROOT_DIR}
+profile=${SBUILD_CHROOT}
+users=${SCHROOT_USER}
+groups=root,sbuild
+root-users=${SCHROOT_USER}
+root-groups=root,sbuild
+preserve-environment=true
+EOF
+
+        mkdir -p "${SCHROOT_DIR}/etc/apt/preferences.d"
+        cat << EOF > "${SCHROOT_DIR}/etc/apt/preferences.d/isar-apt"
+Package: *
+Pin: release n=${DEBDISTRONAME}
+Pin-Priority: 1000
+EOF
+
+        # Prepare mount points
+        cp -rf "${SCHROOT_CONF}/sbuild" "${SBUILD_CONF_DIR}"
+        sbuild_fstab="${SBUILD_CONF_DIR}/fstab"
+
+        fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind 0 0"
+        grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}
+
+        if [ -d ${DL_DIR} ]; then
+            fstab_downloads="${DL_DIR} /downloads none rw,bind 0 0"
+            grep -qxF "${fstab_downloads}" ${sbuild_fstab} || echo "${fstab_downloads}" >> ${sbuild_fstab}
+        fi
+EOSUDO
+}
+
+schroot_delete_configs() {
+    sudo -s <<'EOSUDO'
+        set -e
+        if [ -d "${SBUILD_CONF_DIR}" ]; then
+            rm -rf "${SBUILD_CONF_DIR}"
+        fi
+        rm -f "${SCHROOT_DIR}/etc/apt/preferences.d/isar-apt"
+        rm -f "${SCHROOT_CONF_FILE}"
+        rm -f "${SCHROOT_CONF_FILE_RW}"
+EOSUDO
+}
+
+sbuild_export() {
+    SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
+    VAR=${1}; shift
+    VAR_LINE="'${VAR}' => '${@}',"
+    if [ -s "${SBUILD_CONFIG}" ]; then
+        sed -i -e "\$i\\" -e "${VAR_LINE}" ${SBUILD_CONFIG}
+    else
+        echo "\$build_environment = {" > ${SBUILD_CONFIG}
+        echo "${VAR_LINE}" >> ${SBUILD_CONFIG}
+        echo "};" >> ${SBUILD_CONFIG}
+    fi
+    export SBUILD_CONFIG="${SBUILD_CONFIG}"
+}
+
+schroot_install() {
+    schroot_create_configs
+    APTS="$1"
+
+    distro="${DISTRO}"
+    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+       distro="${HOST_DISTRO}"
+    fi
+
+    deb_dl_dir_import ${SCHROOT_DIR} ${distro}
+    schroot -d / -c ${SBUILD_CHROOT_RW} -u root -- \
+        apt install -y -o Debug::pkgProblemResolver=yes \
+                    --no-install-recommends --download-only ${APTS}
+    deb_dl_dir_export ${SCHROOT_DIR} ${distro}
+
+    schroot -d / -c ${SBUILD_CHROOT_RW} -u root -- \
+        apt install -y -o Debug::pkgProblemResolver=yes \
+                    --no-install-recommends ${APTS}
+    schroot_delete_configs
+}
+
+insert_mounts() {
+    sudo -s <<'EOSUDO'
+        set -e
+        for mp in ${SCHROOT_MOUNTS}; do
+            FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
+            grep -qxF "${FSTAB_LINE}" ${SBUILD_CONF_DIR}/fstab || \
+                echo "${FSTAB_LINE}" >> ${SBUILD_CONF_DIR}/fstab
+        done
+EOSUDO
+}
+
+remove_mounts() {
+    sudo -s <<'EOSUDO'
+        set -e
+        for mp in ${SCHROOT_MOUNTS}; do
+            FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
+            sed -i "\|${FSTAB_LINE}|d" ${SBUILD_CONF_DIR}/fstab
+        done
+EOSUDO
+}
+
+schroot_run() {
+    schroot_create_configs
+    insert_mounts
+    schroot $@
+    remove_mounts
+    schroot_delete_configs
+}
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index cd1c4a64..c1458f05 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -58,6 +58,8 @@ SSTATE_DIR ?= "${TOPDIR}/sstate-cache"
 SSTATE_MANIFESTS = "${TMPDIR}/sstate-control/${DISTRO}-${DISTRO_ARCH}"
 BUILDCHROOT_HOST_DIR = "${DEPLOY_DIR_BUILDCHROOT}-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
 BUILDCHROOT_TARGET_DIR = "${DEPLOY_DIR_BUILDCHROOT}-target/${DISTRO}-${DISTRO_ARCH}"
+SCHROOT_HOST_DIR = "${DEPLOY_DIR}/schroot-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
+SCHROOT_TARGET_DIR = "${DEPLOY_DIR}/schroot-target/${DISTRO}-${DISTRO_ARCH}"
 SDKCHROOT_DIR = "${DEPLOY_DIR_SDKCHROOT}/${DISTRO}-${DISTRO_ARCH}"
 CACHE = "${TMPDIR}/cache"
 KERNEL_FILE ?= "vmlinuz"
-- 
2.20.1

[PATCH v7 05/13] dpkg: Build packages with sbuild

[Uladzimir Bely]

From: Anton Mikanovich <amikan@ilbers.de>

Use previously exported to schroot rootfs to build the package with
sbuild tool. Provide isar-apt as extra repository to be automatically
add to apt sources.

Also added /home/.git-downloads mount while it's used
as git alternates location.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 31 ++++++++++++++++---------------
 meta/classes/dpkg-gbp.bbclass  |  2 ++
 meta/classes/dpkg.bbclass      | 10 ++++++++--
 meta/classes/sbuild.bbclass    |  4 ++--
 4 files changed, 28 insertions(+), 19 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 550bedfc..31d41754 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -4,6 +4,7 @@
 #
 # SPDX-License-Identifier: MIT
 
+inherit sbuild
 inherit buildchroot
 inherit debianize
 inherit terminal
@@ -16,6 +17,8 @@ DEPENDS_append_riscv64 = "${@' crossbuild-essential-riscv64' if d.getVar('ISAR_C
 DEB_BUILD_PROFILES ?= ""
 DEB_BUILD_OPTIONS ?= ""
 
+ISAR_APT_REPO ?= "deb [trusted=yes] file:///home/builder/${PN}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO} ${DEBDISTRONAME} main"
+
 python do_adjust_git() {
     import subprocess
 
@@ -173,6 +176,7 @@ addtask prepare_build after do_patch do_transform_template before do_dpkg_build
 # If Isar recipes depend on each other, they typically need the package
 # deployed to isar-apt
 do_prepare_build[deptask] = "do_deploy_deb"
+do_prepare_build[depends] = "${SCHROOT_DEP}"
 
 BUILDROOT = "${BUILDCHROOT_DIR}/${PP}"
 
@@ -198,6 +202,15 @@ dpkg_undo_mounts() {
     sudo rmdir ${BUILDROOT}
 }
 
+do_prepare_build_append() {
+    # Make a local copy of isar-apt repo that is not affected by other parallel builds
+    mkdir -p ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
+    rm -rf ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/*
+    cp -Rl ${REPO_ISAR_DIR} ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
+}
+
+do_prepare_build[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
+
 # Placeholder for actual dpkg_runbuild() implementation
 dpkg_runbuild() {
     die "This should never be called, overwrite it in your derived class"
@@ -220,14 +233,11 @@ def isar_export_build_settings(d):
     os.environ['DEB_BUILD_PROFILES'] = isar_deb_build_profiles(d)
 
 python do_dpkg_build() {
-    lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
-                             shared=True)
-    bb.build.exec_func("dpkg_do_mounts", d)
+    bb.build.exec_func('schroot_create_configs', d)
     try:
         bb.build.exec_func("dpkg_runbuild", d)
     finally:
-        bb.build.exec_func("dpkg_undo_mounts", d)
-        bb.utils.unlockfile(lock)
+        bb.build.exec_func('schroot_delete_configs', d)
 }
 
 addtask dpkg_build
@@ -259,7 +269,7 @@ python do_dpkg_build_setscene() {
 addtask dpkg_build_setscene
 do_dpkg_build_setscene[dirs] += "${S}/.."
 
-KEEP_INSTALLED_ON_CLEAN ?= "0"
+do_dpkg_build[depends] = "${SCHROOT_DEP}"
 
 CLEANFUNCS += "deb_clean"
 
@@ -269,15 +279,6 @@ deb_clean() {
         for d in ${DEBS}; do
             repo_del_package "${REPO_ISAR_DIR}"/"${DISTRO}" \
                 "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${d}"
-            if [ "${KEEP_INSTALLED_ON_CLEAN}" = "1" ]; then
-                continue;
-            fi
-            package=$(basename "${d}")
-            package_remove="/usr/bin/apt-get remove -y ${package%%_*}"
-            sudo -E chroot ${BUILDCHROOT_DIR} ${package_remove} || true
-            if [ "${BUILDCHROOT_DIR}" != "${BUILDCHROOT_TARGET_DIR}" ]; then
-                    sudo -E chroot ${BUILDCHROOT_TARGET_DIR} ${package_remove} || true
-            fi
         done
     fi
 }
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index 2fc37b2f..7eda9b03 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -11,6 +11,8 @@ PATCHTOOL ?= "git"
 
 GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
 
+SCHROOT_MOUNTS = "${WORKDIR}:${PP} ${GITDIR}:/home/.git-downloads"
+
 dpkg_runbuild_prepend() {
     sh -c "
         cd ${WORKDIR}/${PPS}
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 320102ba..848ffad1 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -39,6 +39,12 @@ dpkg_runbuild() {
     export DEB_BUILD_OPTIONS="${@ isar_deb_build_options(d)}"
     export DEB_BUILD_PROFILES="${@ isar_deb_build_profiles(d)}"
     export PARALLEL_MAKE="${PARALLEL_MAKE}"
-    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
-         /isar/build.sh ${PP}/${PPS} ${PACKAGE_ARCH}
+
+    export SBUILD_CONFIG="${SBUILD_CONFIG}"
+
+    sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
+        --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \
+        --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
+        --debbuildopts="--source-option=-I" \
+        --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
 }
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index e4341a6f..6fff664f 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -31,6 +31,8 @@ SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}"
 SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}"
 SCHROOT_CONF_FILE_RW ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}-rw"
 
+SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
+
 schroot_create_configs() {
     sudo -s <<'EOSUDO'
         set -e
@@ -96,7 +98,6 @@ EOSUDO
 }
 
 sbuild_export() {
-    SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
     VAR=${1}; shift
     VAR_LINE="'${VAR}' => '${@}',"
     if [ -s "${SBUILD_CONFIG}" ]; then
@@ -106,7 +107,6 @@ sbuild_export() {
         echo "${VAR_LINE}" >> ${SBUILD_CONFIG}
         echo "};" >> ${SBUILD_CONFIG}
     fi
-    export SBUILD_CONFIG="${SBUILD_CONFIG}"
 }
 
 schroot_install() {
-- 
2.20.1

[PATCH v7 06/13] sbuild: support of DEB_BUILD_PROFILES

[Uladzimir Bely]

In Isar packages may use their own profiles. For example, linux-custom.inc
introduces 'nolibcdev' profile that should be passed to dpkg-buildpackage
in both cross and native builds.

By default, sbuild for cross-building uses passes "cross,nocheck" options
to dpkg-buildpackage and ignores DEB_BUILD_PROFILES environment variable.

This change makes sbuild use custom profiles even in cross-build mode.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 848ffad1..5511de64 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -40,10 +40,18 @@ dpkg_runbuild() {
     export DEB_BUILD_PROFILES="${@ isar_deb_build_profiles(d)}"
     export PARALLEL_MAKE="${PARALLEL_MAKE}"
 
+    profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 | cut -d "'" -f 4)
+    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+        profiles="${profiles} cross nocheck"
+    fi
+    if [ ! -z "$profiles" ]; then
+        profiles=$(echo --profiles="$profiles" | sed -e 's/ \+/,/g')
+    fi
+
     export SBUILD_CONFIG="${SBUILD_CONFIG}"
 
     sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
-        --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \
+        --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
         --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
         --debbuildopts="--source-option=-I" \
         --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
-- 
2.20.1

[PATCH v7 07/13] sbuild: support of shell exports from dpkg_runbuild_prepend

[Uladzimir Bely]

Many of recipes often use shell exports done in dpkg_run_prepend.

While sbuild is performed in isolated environment, we need a way to pass
these exported variables to it. This is done by storing environment just
before dpkg_runbuild (after prepare_build) and finding just before
the actual build what has been changed or added.

This patch allows using shell exports, but warning will be produced.
It's recommended to modify the code (in downstreams) and replace shell
exports by other methods, like using templates.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 5511de64..82a722df 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -32,6 +32,12 @@ do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
 
 addtask devshell after do_install_builddeps
 
+DPKG_PREBUILD_ENV_FILE="${WORKDIR}/dpkg_prebuild.env"
+
+do_prepare_build_append() {
+    env > ${DPKG_PREBUILD_ENV_FILE}
+}
+
 # Build package from sources using build script
 dpkg_runbuild() {
     E="${@ isar_export_proxies(d)}"
@@ -40,6 +46,27 @@ dpkg_runbuild() {
     export DEB_BUILD_PROFILES="${@ isar_deb_build_profiles(d)}"
     export PARALLEL_MAKE="${PARALLEL_MAKE}"
 
+    env | while read -r line; do
+        # Filter the same lines
+        grep -q "^${line}" ${DPKG_PREBUILD_ENV_FILE} && continue
+        # Filter some standard variables
+        echo ${line} | grep -q "^HOME=" && continue
+        echo ${line} | grep -q "^PWD=" && continue
+
+        var=$(echo "${line}" | cut -d '=' -f1)
+        value=$(echo "${line}" | cut -d '=' -f2-)
+        sbuild_export $var "$value"
+
+        # Don't warn some variables
+        [ "${var}" = "PARALLEL_MAKE" ] && continue
+        [ "${var}" = "CCACHE_DIR" ] && continue
+        [ "${var}" = "PATH_PREPEND" ] && continue
+        [ "${var}" = "DEB_BUILD_OPTIONS" ] && continue
+        [ "${var}" = "DEB_BUILD_PROFILES" ] && continue
+
+        bbwarn "Export of '${line}' detected, please migrate to templates"
+    done
+
     profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 | cut -d "'" -f 4)
     if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
         profiles="${profiles} cross nocheck"
-- 
2.20.1

[PATCH v7 08/13] dpkg: Remove builddeps install task.

[Uladzimir Bely]

While builddeps are now handled interlnally by sbuild, we don't need
a task to preinstall them in buildchroot.

But for the second local (base-apt) build we need a way to keep
all dependencies in form of .deb files in DL_DIR. This is done
by executing additional commands in sbuild that copy them to/from
an externally mounted folder.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass | 47 +++++++++++++++++----------------------
 1 file changed, 20 insertions(+), 27 deletions(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 82a722df..4cf405ad 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -5,33 +5,6 @@ inherit dpkg-base
 
 PACKAGE_ARCH ?= "${DISTRO_ARCH}"
 
-# Install build dependencies for package
-do_install_builddeps() {
-    dpkg_do_mounts
-    E="${@ isar_export_proxies(d)}"
-    export DEB_BUILD_OPTIONS="${@ isar_deb_build_options(d)}"
-    export DEB_BUILD_PROFILES="${@ isar_deb_build_profiles(d)}"
-    distro="${DISTRO}"
-    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
-        distro="${HOST_DISTRO}"
-    fi
-
-    deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
-        ${PP}/${PPS} ${PACKAGE_ARCH} --download-only
-    deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
-        ${PP}/${PPS} ${PACKAGE_ARCH}
-    dpkg_undo_mounts
-}
-
-addtask install_builddeps after do_prepare_build before do_dpkg_build
-do_install_builddeps[depends] += "isar-apt:do_cache_config"
-# apt and reprepro may not run in parallel, acquire the Isar lock
-do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
-
-addtask devshell after do_install_builddeps
-
 DPKG_PREBUILD_ENV_FILE="${WORKDIR}/dpkg_prebuild.env"
 
 do_prepare_build_append() {
@@ -67,6 +40,21 @@ dpkg_runbuild() {
         bbwarn "Export of '${line}' detected, please migrate to templates"
     done
 
+    distro="${DISTRO}"
+    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+        distro="${HOST_DISTRO}"
+    fi
+
+    deb_dl_dir_import "${WORKDIR}/rootfs" "${distro}"
+
+    deb_dir="/var/cache/apt/archives/"
+    ext_deb_dir="/home/builder/${PN}/rootfs/${deb_dir}"
+
+    ( flock 9
+        grep -qxF '$apt_keep_downloaded_packages = 1;' ${SCHROOT_USER_HOME}/.sbuildrc ||
+            echo '$apt_keep_downloaded_packages = 1;' >> ${SCHROOT_USER_HOME}/.sbuildrc
+    ) 9>"${TMPDIR}/sbuildrc.lock"
+
     profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 | cut -d "'" -f 4)
     if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
         profiles="${profiles} cross nocheck"
@@ -80,6 +68,11 @@ dpkg_runbuild() {
     sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
         --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
         --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
+        --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
+        --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
+        --finished-build-commands="cp -n --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
         --debbuildopts="--source-option=-I" \
         --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
+
+    deb_dl_dir_export "${WORKDIR}/rootfs" "${distro}"
 }
-- 
2.20.1

[PATCH v7 09/13] sbuild: add ccache support

[Uladzimir Bely]

This adds ccache support for custom packages in the same manner
as it was done previously for buildchroot-based build, by using
USE_CCACHE variable in local.conf or per-recipe.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass                     |  4 ++++
 meta/classes/sbuild.bbclass                   | 21 +++++++++++++++++++
 .../sbuild-chroot/sbuild-chroot.inc           |  1 +
 3 files changed, 26 insertions(+)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 4cf405ad..3cd40ba8 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -55,6 +55,10 @@ dpkg_runbuild() {
             echo '$apt_keep_downloaded_packages = 1;' >> ${SCHROOT_USER_HOME}/.sbuildrc
     ) 9>"${TMPDIR}/sbuildrc.lock"
 
+    if [ ${USE_CCACHE} -eq 1 ]; then
+        schroot_configure_ccache
+    fi
+
     profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 | cut -d "'" -f 4)
     if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
         profiles="${profiles} cross nocheck"
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 6fff664f..9896b772 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -158,3 +158,24 @@ schroot_run() {
     remove_mounts
     schroot_delete_configs
 }
+
+schroot_configure_ccache() {
+    sudo -s <<'EOSUDO'
+        set -e
+
+        sbuild_fstab="${SBUILD_CONF_DIR}/fstab"
+
+        install --group=sbuild --mode=2775 -d ${CCACHE_DIR}
+        fstab_ccachedir="${CCACHE_DIR} /ccache none rw,bind 0 0"
+        grep -qxF "${fstab_ccachedir}" ${sbuild_fstab} || echo "${fstab_ccachedir}" >> ${sbuild_fstab}
+
+        cat << END > ${CCACHE_DIR}/sbuild-setup
+#!/bin/sh
+export PATH="\$PATH_PREPEND:\$PATH"
+exec "\$@"
+END
+        chmod a+rx ${CCACHE_DIR}/sbuild-setup
+
+        echo "command-prefix=/ccache/sbuild-setup" >> "${SCHROOT_CONF_FILE}"
+EOSUDO
+}
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
index 177a8a6d..74811e22 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -14,6 +14,7 @@ SBUILD_CHROOT_PREINSTALL_COMMON = " \
     fakeroot \
     build-essential \
     debhelper \
+    ccache \
 "
 
 SBUILD_CHROOT_PREINSTALL ?= " \
-- 
2.20.1

[PATCH v7 10/13] dpkg-base: Switch devshell to use schroot

[Uladzimir Bely]

While packages are now build in schroot with sbuild, devshell
task should make terminal open inside sbuild environment.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-base.bbclass                | 23 ++++++++++++++-----
 meta/classes/sbuild.bbclass                   |  3 +++
 .../sbuild-chroot/sbuild-chroot.inc           |  2 ++
 3 files changed, 22 insertions(+), 6 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 31d41754..238a0bbd 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -296,19 +296,30 @@ do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
 do_deploy_deb[dirs] = "${S}"
 
 python do_devshell() {
-    bb.build.exec_func('dpkg_do_mounts', d)
+    bb.build.exec_func('schroot_create_configs', d)
 
     isar_export_proxies(d)
     isar_export_ccache(d)
     isar_export_build_settings(d)
 
-    buildchroot = d.getVar('BUILDCHROOT_DIR')
+    schroot = d.getVar('SBUILD_CHROOT')
+    isar_apt = d.getVar('ISAR_APT_REPO')
     pp_pps = os.path.join(d.getVar('PP'), d.getVar('PPS'))
-    # the PATH variable is not forwarded by sudo -E.
-    termcmd = "sudo -E chroot {0} sh -c 'cd {1}; export PATH=$PATH_PREPEND:$PATH; $SHELL -i'"
-    oe_terminal(termcmd.format(buildchroot, pp_pps), "Isar devshell", d)
 
-    bb.build.exec_func('dpkg_undo_mounts', d)
+    install_deps = ":" if d.getVar('BB_CURRENTTASK') == "devshell_nodeps" else "mk-build-deps -i -t \
+        \"apt-get -y -q -o Debug::pkgProblemResolver=yes --no-install-recommends --allow-downgrades\" \
+        debian/control"
+
+    termcmd = "schroot -d / -c {0} -u root -- sh -c ' \
+        cd {1}; \
+        echo {2} > /etc/apt/sources.list.d/isar_apt.list; \
+        apt-get -y -q update; \
+        {3}; \
+        $SHELL -i \
+    '"
+    oe_terminal(termcmd.format(schroot, pp_pps, isar_apt, install_deps), "Isar devshell", d)
+
+    bb.build.exec_func('schroot_delete_configs', d)
 }
 
 addtask devshell after do_prepare_build
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 9896b772..eb358c71 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -78,6 +78,9 @@ EOF
         fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind 0 0"
         grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}
 
+        fstab_pkgdir="${WORKDIR} /home/builder/${PN} none rw,bind 0 0"
+        grep -qxF "${fstab_pkgdir}" ${sbuild_fstab} || echo "${fstab_pkgdir}" >> ${sbuild_fstab}
+
         if [ -d ${DL_DIR} ]; then
             fstab_downloads="${DL_DIR} /downloads none rw,bind 0 0"
             grep -qxF "${fstab_downloads}" ${sbuild_fstab} || echo "${fstab_downloads}" >> ${sbuild_fstab}
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
index 74811e22..4688da1d 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -15,6 +15,8 @@ SBUILD_CHROOT_PREINSTALL_COMMON = " \
     build-essential \
     debhelper \
     ccache \
+    devscripts \
+    equivs \
 "
 
 SBUILD_CHROOT_PREINSTALL ?= " \
-- 
2.20.1

[PATCH v7 11/13] dpkg-base: Switch apt_fetch and apt_unpack to use schroot

[Uladzimir Bely]

This moves downloading and unpacking deb-src to schroot environment.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 25 +++++++++----------------
 1 file changed, 9 insertions(+), 16 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 238a0bbd..7220f64f 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -101,19 +101,12 @@ python() {
 }
 
 do_apt_fetch() {
-    dpkg_do_mounts
-    E="${@ isar_export_proxies(d)}"
-    sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update \
-        -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
-        -o Dir::Etc::SourceParts="-" \
-        -o APT::Get::List-Cleanup="0"
-
+    schroot_create_configs
     for uri in "${SRC_APT}"; do
-        sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+        schroot -d / -c ${SBUILD_CHROOT} -- \
             sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only-source source "$2"' my_script "${DISTRO}" "${uri}"
     done
-
-    dpkg_undo_mounts
+    schroot_delete_configs
 }
 
 addtask apt_fetch after do_unpack before do_apt_unpack
@@ -122,13 +115,14 @@ do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
 # Add dependency from the correct buildchroot: host or target
 do_apt_fetch[depends] = "${BUILDCHROOT_DEP}"
 
+# Add dependency from the correct schroot: host or target
+do_apt_fetch[depends] += "${SCHROOT_DEP}"
+
 do_apt_unpack() {
     rm -rf ${S}
-    dpkg_do_mounts
-    E="${@ isar_export_proxies(d)}"
-
+    schroot_create_configs
     for uri in "${SRC_APT}"; do
-        sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+        schroot -d / -c ${SBUILD_CHROOT} -- \
             sh -c ' \
                 set -e
                 dscfile="$(apt-get -y -qq --print-uris --only-source source "${2}" | cut -d " " -f2 | grep -E "*.dsc")"
@@ -137,8 +131,7 @@ do_apt_unpack() {
                 dpkg-source -x "${dscfile}" "${PPS}"' \
                     my_script "${DISTRO}" "${uri}"
     done
-
-    dpkg_undo_mounts
+    schroot_delete_configs
 }
 
 addtask apt_unpack after do_apt_fetch before do_patch
-- 
2.20.1

[PATCH v7 12/13] doc: Add sbuild-related documentation

[Uladzimir Bely]

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 doc/user_manual.md | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index df73ef89..966e7125 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -85,6 +85,24 @@ apt install \
   pristine-tar
 ```
 
+Additional setup is required since `sbuild` is now used for package build.
+Install the following packages:
+```
+apt install \
+  sbuild \
+  schroot
+```
+Also, user who runs isar should be added to `sbuild` group.
+
+**NOTE:** sbuild version (<=0.78.1) packaged in Debian Buster doesn't support
+`$apt_keep_downloaded_packages` option which is required in Isar for
+populating `${DL_DIR}/deb`. So, host `sbuild` in this case should be manually
+upgraded to >=0.81.2 version from Debian Bullseye.
+
+```
+sudo gpasswd -a <username> sbuild
+```
+
 If your host is >= buster, also install the following package.
 ```
 apt install python3-distutils
-- 
2.20.1

[PATCH v7 13/13] sbuild: Use .dsc file instead of source directory.

[Uladzimir Bely]

This makes build more Debian-like and avoids additional execution of
"dh clean" by sbuild before dependencies resolved and build started.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 3cd40ba8..19bd582d 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -69,6 +69,10 @@ dpkg_runbuild() {
 
     export SBUILD_CONFIG="${SBUILD_CONFIG}"
 
+    # Create a .dsc file from source directory to use it with sbuild
+    dpkg-source -q -b ${WORKDIR}/${PPS}
+    DSC=$(head -n1 ${WORKDIR}/${PPS}/debian/changelog | awk '{gsub(/[()]/,""); printf "%s_%s.dsc", $1, $2}')
+
     sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
         --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
         --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
@@ -76,7 +80,7 @@ dpkg_runbuild() {
         --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
         --finished-build-commands="cp -n --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
         --debbuildopts="--source-option=-I" \
-        --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
+        --build-dir=${WORKDIR} --dist="isar" ${WORKDIR}/${DSC}
 
     deb_dl_dir_export "${WORKDIR}/rootfs" "${distro}"
 }
-- 
2.20.1

RE: [PATCH v7 06/13] sbuild: support of DEB_BUILD_PROFILES

[Moessbauer, Felix]

Hi,

> -----Original Message-----
> From: isar-users@googlegroups.com <isar-users@googlegroups.com> On
> Behalf Of Uladzimir Bely
> Sent: Sunday, February 13, 2022 8:45 AM
> To: isar-users@googlegroups.com
> Subject: [PATCH v7 06/13] sbuild: support of DEB_BUILD_PROFILES
> 
> In Isar packages may use their own profiles. For example, linux-custom.inc
> introduces 'nolibcdev' profile that should be passed to dpkg-buildpackage in
> both cross and native builds.
> 
> By default, sbuild for cross-building uses passes "cross,nocheck" options to
> dpkg-buildpackage and ignores DEB_BUILD_PROFILES environment variable.

There is a typo (uses passes).

> 
> This change makes sbuild use custom profiles even in cross-build mode.
> 
> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> ---
>  meta/classes/dpkg.bbclass | 10 +++++++++-
>  1 file changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass index
> 848ffad1..5511de64 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -40,10 +40,18 @@ dpkg_runbuild() {
>      export DEB_BUILD_PROFILES="${@ isar_deb_build_profiles(d)}"
>      export PARALLEL_MAKE="${PARALLEL_MAKE}"
> 
> +    profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 | cut -d
> "'" -f 4)
> +    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
> +        profiles="${profiles} cross nocheck"
> +    fi
> +    if [ ! -z "$profiles" ]; then
> +        profiles=$(echo --profiles="$profiles" | sed -e 's/ \+/,/g')
> +    fi
> +
>      export SBUILD_CONFIG="${SBUILD_CONFIG}"
> 
>      sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
> -        --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \
> +        --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles}

As we handle the DEB_BUILD_PROFILES exclusively in ISAR anyways, can't we simplify this code by always adding --profiles="${@ isar_deb_build_profiles(d)}" and get rid of the export?
Otherwise we have to maintain the "cross nocheck" logic for cross-compiling at two different locations.

Felix


> + \
>          --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
>          --debbuildopts="--source-option=-I" \
>          --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
> --
> 2.20.1
> 
> --
> You received this message because you are subscribed to the Google Groups
> "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to isar-users+unsubscribe@googlegroups.com.
> To view this discussion on the web visit
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.g
> oogle.com%2Fd%2Fmsgid%2Fisar-users%2F20220213074459.24200-7-
> ubely%2540ilbers.de&amp;data=04%7C01%7Cfelix.moessbauer%40siemens.co
> m%7C9a465e3edb52484113a108d9eec4bf92%7C38ae3bcd95794fd4addab42e1
> 495d55a%7C1%7C0%7C637803351819682721%7CUnknown%7CTWFpbGZsb3d8
> eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%
> 7C3000&amp;sdata=Qq0ecQNOACm2ILez0sEG1A4qlyvrZ04kRT8dxkc4w2c%3D&
> amp;reserved=0.

Re: [PATCH v7 06/13] sbuild: support of DEB_BUILD_PROFILES

[Uladzimir Bely]

In the email from Sunday, 13 February 2022 19:04:48 +03 user Moessbauer, Felix 
wrote:
> Hi,
> 
> > -----Original Message-----
> > From: isar-users@googlegroups.com <isar-users@googlegroups.com> On
> > Behalf Of Uladzimir Bely
> > Sent: Sunday, February 13, 2022 8:45 AM
> > To: isar-users@googlegroups.com
> > Subject: [PATCH v7 06/13] sbuild: support of DEB_BUILD_PROFILES
> > 
> > In Isar packages may use their own profiles. For example, linux-custom.inc
> > introduces 'nolibcdev' profile that should be passed to dpkg-buildpackage
> > in both cross and native builds.
> > 
> > By default, sbuild for cross-building uses passes "cross,nocheck" options
> > to dpkg-buildpackage and ignores DEB_BUILD_PROFILES environment variable.
> There is a typo (uses passes).
> 
> > This change makes sbuild use custom profiles even in cross-build mode.
> > 
> > Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> > ---
> > 
> >  meta/classes/dpkg.bbclass | 10 +++++++++-
> >  1 file changed, 9 insertions(+), 1 deletion(-)
> > 
> > diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass index
> > 848ffad1..5511de64 100644
> > --- a/meta/classes/dpkg.bbclass
> > +++ b/meta/classes/dpkg.bbclass
> > @@ -40,10 +40,18 @@ dpkg_runbuild() {
> > 
> >      export DEB_BUILD_PROFILES="${@ isar_deb_build_profiles(d)}"
> >      export PARALLEL_MAKE="${PARALLEL_MAKE}"
> > 
> > +    profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 |
> > cut -d "'" -f 4)
> > +    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
> > +        profiles="${profiles} cross nocheck"
> > +    fi
> > +    if [ ! -z "$profiles" ]; then
> > +        profiles=$(echo --profiles="$profiles" | sed -e 's/ \+/,/g')
> > +    fi
> > +
> > 
> >      export SBUILD_CONFIG="${SBUILD_CONFIG}"
> >      
> >      sbuild -A -n -c ${SBUILD_CHROOT}
> >      --extra-repository="${ISAR_APT_REPO}" \
> > 
> > -        --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \
> > +        --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles}
> 
> As we handle the DEB_BUILD_PROFILES exclusively in ISAR anyways, can't we
> simplify this code by always adding --profiles="${@
> isar_deb_build_profiles(d)}" and get rid of the export? Otherwise we have
> to maintain the "cross nocheck" logic for cross-compiling at two different
> locations.
> 

I need to check it. In isar_deb_build_profiles() we just add " cross" to 
profiles list in case of cross-build. Probably, sbuild also requires " 
nocheck" to be passed.
Anyway, sbuild accepts comma-separated list, while environment variable is a 
space-separated one. So, some handling of this will be still required.

> Felix
> 
> > + \
> > 
> >          --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
> >          --debbuildopts="--source-option=-I" \
> >          --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
> > 
> > --
> > 2.20.1
> > 
> > --
> > You received this message because you are subscribed to the Google Groups
> > "isar-users" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to isar-users+unsubscribe@googlegroups.com.
> > To view this discussion on the web visit
> > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.g
> > oogle.com%2Fd%2Fmsgid%2Fisar-users%2F20220213074459.24200-7-
> > ubely%2540ilbers.de&amp;data=04%7C01%7Cfelix.moessbauer%40siemens.co
> > m%7C9a465e3edb52484113a108d9eec4bf92%7C38ae3bcd95794fd4addab42e1
> > 495d55a%7C1%7C0%7C637803351819682721%7CUnknown%7CTWFpbGZsb3d8
> > eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%
> > 7C3000&amp;sdata=Qq0ecQNOACm2ILez0sEG1A4qlyvrZ04kRT8dxkc4w2c%3D&
> > amp;reserved=0.


-- 
Uladzimir Bely