[PATCH v12 00/21] Sbuild/Schroot migration

[PATCH v12 00/21] Sbuild/Schroot migration

[Anton Mikanovich]

This patchset integrates sbuild/schroot tools into Isar build system.

Base schroot images are created with sbuild-chroot-target or, in case
of cross-build, sbuild-chroot-host recipes. These images are based on
isar-bootstrap ones, but include some build-related stuff preinstalled.

To use these changes you need to have sbuild and schroot installed and
configured. User who runs build should be added to sbuild group.

Currently, sbuild-related stuff is supported in latest kas.

If Gilab CI is used, the following configuration changes are required:
- external dir for schroot overlay should be set in the config
/etc/gitlab-runner/config.toml:

volumes = ["/path/to/overlay:/var/lib/schroot/union/overlay"]

The patchset was tested with 'meta-iot2050' and 'xenomai-images'
downstreams in combination with kas image mentioned above.

Current limitations:
- parallel building of several packages requires more free space in
comparison with buildchroot-based architecture. This happens due
the sbuild architecture which uses some basic layer (common for all
packages) and per-package separate layer (where builddeps are downloaded
and installed).

Changes sinse v11:
- Rebase on current next.

Changes sinse v10:
- Rebase on current next.

Changes sinse v9:
- Add caching improvement patch;
- Remove per-package deb artifacts dir after usage to reduce disk space;
- Improve schroot cleanup on failures.

Changes since v8:
- Added patch with support of package versions with epochs

Changes since v7:
- fixed work with proxy by adding some environment filters in sbuild;
- DEB_BUILD_PROFILES support simplified and improved
- Fixed gbp-based recipes rebuilding (get back --git-ignore-new option
previously removed)
- fixed base-apt caching for packages and source packages temporary
installed in sbuild isolated environment
- fixed rare build error related to ccache setup (added locking)

Changes since v6:
- added patch that switches sbuild from using source directory to using
.dsc file. The file is generated before sbuild call by `dpkg-source`.

Changes since v5:
- warning about shell exports in dpkg_runbuild_prepend. This should
avoided be reworked to templates.

Changes since v4:
- consider shell exports done in dpkg_runbuild_prepend and pass them
to sbuild environment;
- fixed ccache work.

Changes since v3:
- dpkg_do_mounts() and dpkg_undo_mounts() are not removed for downstreams
compatibility;
- dpkg_build_export is used for adjusting sbuild environment;
- DEB_BUILD_PROFILES support for sbuild is fixed in case of cross-build.

Changes since v2:
- patches reworked/squashed for easier reading and understanding;
- fixed building foreigh architectures with kas-docker;
- implemented support of ccache;
- fixed devshell and devshell_nodeps.

Changes since v1:
- parallel builds with different BUILD_DIR are supported;
- parallel multiconfig targets in one build are supported;
- per-task schroot configuration in /etc/schroot/ is now used;
- patchset now passes Jenkins CI (so patches changes RFC => PATCH).

Anton Mikanovich (3):
  dpkg: Build packages with sbuild
  dpkg-base: Cleanup on schroot fail
  sbuild: Add changelog entry

Felix Moessbauer (4):
  fix: support build of packages with epoch version
  always create apt-cache dirs in deb_dl_dir_import
  avoid absolute SCHROOT_* paths to improve caching
  locate generated binary package in WORKDIR

Uladzimir Bely (14):
  dpkg-gbp: Use separate command to export tarball
  dpkg-gbp: Use host tools for dsc preparation
  sbuild: Add recipes for host and target rootfs to run sbuild
  sbuild: Introduce a class for another build method
  sbuild: Support of DEB_BUILD_PROFILES
  sbuild: Support of shell exports from dpkg_runbuild_prepend
  dpkg: Remove builddeps install task
  sbuild: Add ccache support
  dpkg-base: Switch devshell to use schroot
  dpkg-base: Switch apt_fetch and apt_unpack to use schroot
  doc: Add sbuild-related documentation
  sbuild: Use .dsc file instead of source directory
  sbuild: Fixed proxy support
  sbuild: Fix debsrc_download for packages dependencies

 RECIPE-API-CHANGELOG.md                       |  13 ++
 doc/user_manual.md                            |  22 ++-
 meta/classes/deb-dl-dir.bbclass               |   6 +-
 meta/classes/dpkg-base.bbclass                |  90 ++++++----
 meta/classes/dpkg-gbp.bbclass                 |  26 +--
 meta/classes/dpkg.bbclass                     | 109 +++++++++---
 meta/classes/sbuild.bbclass                   | 159 ++++++++++++++++++
 meta/conf/bitbake.conf                        |   2 +
 .../sbuild-chroot/sbuild-chroot-host.bb       |  13 ++
 .../sbuild-chroot/sbuild-chroot-target.bb     |  10 ++
 .../sbuild-chroot/sbuild-chroot.inc           |  39 +++++
 11 files changed, 408 insertions(+), 81 deletions(-)
 create mode 100644 meta/classes/sbuild.bbclass
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc

-- 
2.17.1

[PATCH v12 01/21] dpkg-gbp: Use separate command to export tarball

[Anton Mikanovich]

From: Uladzimir Bely <ubely@ilbers.de>

We don't actually build the package with gbp, but only prepare it for
the building with external dpkg-buildpackage command. In case there is
no need in real build we can perform only export which will produce
upstream tarball to pass for later building.
This allows to use any build tool for later processing.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-gbp.bbclass | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index d956e8c..6129d73 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -30,5 +30,9 @@ do_install_builddeps_append() {
 }
 
 dpkg_runbuild_prepend() {
-    export GBP_PREFIX="gbp buildpackage --git-ignore-new ${GBP_EXTRA_OPTIONS} --git-builder="
+    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+        sh -c "cd ${PP}/${PPS} && gbp buildpackage --git-ignore-new --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}"
+    # NOTE: `buildpackage --git-builder=/bin/true --git-pristine-tar` is used
+    # for compatibility with gbp version froms debian-stretch. In newer distros
+    # it's possible to use a subcommand `export-orig --pristine-tar`
 }
-- 
2.17.1

[PATCH v12 02/21] dpkg-gbp: Use host tools for dsc preparation

[Anton Mikanovich]

From: Uladzimir Bely <ubely@ilbers.de>

Instead of preinstalling gbp-related packages in sbuild chroot
we will use their host versions in order to keep sbuild environment
as clean as possible.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 doc/user_manual.md            |  4 +++-
 meta/classes/dpkg-gbp.bbclass | 24 ++++--------------------
 2 files changed, 7 insertions(+), 21 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 02874b6..61834af 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -81,7 +81,9 @@ apt install \
   qemu \
   qemu-user-static \
   reprepro \
-  sudo
+  sudo \
+  git-buildpackage \
+  pristine-tar
 ```
 
 If your host is >= buster, also install the following package.
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index 6129d73..2b0b818 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -9,29 +9,13 @@ S = "${WORKDIR}/git"
 
 PATCHTOOL ?= "git"
 
-GBP_DEPENDS ?= "git-buildpackage pristine-tar"
 GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
 
-do_install_builddeps_append() {
-    dpkg_do_mounts
-    distro="${DISTRO}"
-    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
-       distro="${HOST_DISTRO}"
-    fi
-    deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} \
-        apt-get install -y -o Debug::pkgProblemResolver=yes \
-                        --no-install-recommends --download-only ${GBP_DEPENDS}
-    deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} \
-        apt-get install -y -o Debug::pkgProblemResolver=yes \
-                        --no-install-recommends ${GBP_DEPENDS}
-    dpkg_undo_mounts
-}
-
 dpkg_runbuild_prepend() {
-    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
-        sh -c "cd ${PP}/${PPS} && gbp buildpackage --git-ignore-new --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}"
+    sh -c "
+        cd ${WORKDIR}/${PPS}
+        gbp buildpackage --git-ignore-new --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}
+    "
     # NOTE: `buildpackage --git-builder=/bin/true --git-pristine-tar` is used
     # for compatibility with gbp version froms debian-stretch. In newer distros
     # it's possible to use a subcommand `export-orig --pristine-tar`
-- 
2.17.1

[PATCH v12 03/21] sbuild: Add recipes for host and target rootfs to run sbuild

[Anton Mikanovich]

From: Uladzimir Bely <ubely@ilbers.de>

Similar to buildchroot, we need a separate rootfs to be used
for schroot.

It's based on bootstrapped rootfs, but includes several
common build-related packages.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 .../sbuild-chroot/sbuild-chroot-host.bb       | 13 +++++++
 .../sbuild-chroot/sbuild-chroot-target.bb     | 10 ++++++
 .../sbuild-chroot/sbuild-chroot.inc           | 36 +++++++++++++++++++
 3 files changed, 59 insertions(+)
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc

diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
new file mode 100644
index 0000000..aa82846
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
@@ -0,0 +1,13 @@
+# Root filesystem for packages building
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2021 ilbers GmbH
+
+DESCRIPTION = "Isar sbuild/schroot filesystem for host"
+
+SBUILD_VARIANT = "host"
+
+require sbuild-chroot.inc
+
+ROOTFS_ARCH = "${HOST_ARCH}"
+ROOTFS_DISTRO = "${HOST_DISTRO}"
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
new file mode 100644
index 0000000..d75d783
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
@@ -0,0 +1,10 @@
+# Root filesystem for packages building
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2021 ilbers GmbH
+
+DESCRIPTION = "Isar sbuild/schroot filesystem for target"
+
+SBUILD_VARIANT = "target"
+
+require sbuild-chroot.inc
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
new file mode 100644
index 0000000..177a8a6
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -0,0 +1,36 @@
+# Common part for build chroot filesystem.
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2021 ilbers GmbH
+
+LICENSE = "gpl-2.0"
+LIC_FILES_CHKSUM = "file://${LAYERDIR_core}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
+
+PV = "1.0"
+
+inherit rootfs
+
+SBUILD_CHROOT_PREINSTALL_COMMON = " \
+    fakeroot \
+    build-essential \
+    debhelper \
+"
+
+SBUILD_CHROOT_PREINSTALL ?= " \
+    ${SBUILD_CHROOT_PREINSTALL_COMMON} \
+"
+
+SBUILD_CHROOT_DIR = "${WORKDIR}/rootfs"
+ROOTFSDIR = "${SBUILD_CHROOT_DIR}"
+ROOTFS_PACKAGES = "${SBUILD_CHROOT_PREINSTALL}"
+
+# We don't need /etc/apt/sources.list.d/isar-apt.list' while it's handled by sbuild
+ROOTFS_CONFIGURE_COMMAND_remove = "rootfs_configure_isar_apt"
+
+DEPLOY_SCHROOT = "${@d.getVar('SCHROOT_' + d.getVar('SBUILD_VARIANT').upper() + '_DIR')}"
+
+do_sbuildchroot_deploy[dirs] = "${DEPLOY_DIR}/schroot-${SBUILD_VARIANT}"
+do_sbuildchroot_deploy() {
+    ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_SCHROOT}"
+}
+addtask sbuildchroot_deploy before do_build after do_rootfs
-- 
2.17.1

[PATCH v12 04/21] sbuild: Introduce a class for another build method

[Anton Mikanovich]

From: Uladzimir Bely <ubely@ilbers.de>

This also adds mounts for base-apt inside schroot and adds import/export
deb files to/from schroot. So that it becomes possible to run second
 `cached` build from local base-apt repo.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/sbuild.bbclass | 108 ++++++++++++++++++++++++++++++++++++
 meta/conf/bitbake.conf      |   2 +
 2 files changed, 110 insertions(+)
 create mode 100644 meta/classes/sbuild.bbclass

diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
new file mode 100644
index 0000000..b9fc6a0
--- /dev/null
+++ b/meta/classes/sbuild.bbclass
@@ -0,0 +1,108 @@
+# This software is a part of ISAR.
+# Copyright (C) 2021 ilbers GmbH
+
+SCHROOT_CONF ?= "/etc/schroot"
+
+SCHROOT_MOUNTS ?= ""
+
+python __anonymous() {
+    import pwd
+    d.setVar('SCHROOT_USER', pwd.getpwuid(os.geteuid()).pw_name)
+    d.setVar('SCHROOT_USER_HOME', pwd.getpwuid(os.geteuid()).pw_dir)
+
+    mode = d.getVar('ISAR_CROSS_COMPILE', True)
+    distro_arch = d.getVar('DISTRO_ARCH')
+    if mode == "0" or d.getVar('HOST_ARCH') ==  distro_arch or \
+       (d.getVar('HOST_DISTRO') == "debian-stretch" and distro_arch == "i386"):
+        d.setVar('SBUILD_HOST_ARCH', distro_arch)
+        d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_TARGET_DIR'))
+        dep = "sbuild-chroot-target:do_build"
+    else:
+        d.setVar('SBUILD_HOST_ARCH', d.getVar('HOST_ARCH'))
+        d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_HOST_DIR'))
+        dep = "sbuild-chroot-host:do_build"
+    d.setVar('SCHROOT_DEP', dep)
+}
+
+SBUILD_CHROOT ?= "${DEBDISTRONAME}-${SCHROOT_USER}-${@os.getpid()}"
+
+SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}"
+SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}"
+
+schroot_create_configs() {
+    sudo -s <<'EOSUDO'
+        set -e
+
+        cat << EOF > "${SCHROOT_CONF_FILE}"
+[${SBUILD_CHROOT}]
+type=directory
+directory=${SCHROOT_DIR}
+profile=${SBUILD_CHROOT}
+users=${SCHROOT_USER}
+groups=root,sbuild
+root-users=${SCHROOT_USER}
+root-groups=root,sbuild
+source-root-users=${SCHROOT_USER}
+source-root-groups=root,sbuild
+union-type=overlay
+preserve-environment=true
+EOF
+
+        # Prepare mount points
+        cp -rf "${SCHROOT_CONF}/sbuild" "${SBUILD_CONF_DIR}"
+        sbuild_fstab="${SBUILD_CONF_DIR}/fstab"
+
+        fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind 0 0"
+        grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}
+
+        if [ -d ${DL_DIR} ]; then
+            fstab_downloads="${DL_DIR} /downloads none rw,bind 0 0"
+            grep -qxF "${fstab_downloads}" ${sbuild_fstab} || echo "${fstab_downloads}" >> ${sbuild_fstab}
+        fi
+EOSUDO
+}
+
+schroot_delete_configs() {
+    sudo -s <<'EOSUDO'
+        set -e
+        if [ -d "${SBUILD_CONF_DIR}" ]; then
+            rm -rf "${SBUILD_CONF_DIR}"
+        fi
+        rm -f "${SCHROOT_CONF_FILE}"
+EOSUDO
+}
+
+sbuild_export() {
+    SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
+    VAR=${1}; shift
+    VAR_LINE="'${VAR}' => '${@}',"
+    if [ -s "${SBUILD_CONFIG}" ]; then
+        sed -i -e "\$i\\" -e "${VAR_LINE}" ${SBUILD_CONFIG}
+    else
+        echo "\$build_environment = {" > ${SBUILD_CONFIG}
+        echo "${VAR_LINE}" >> ${SBUILD_CONFIG}
+        echo "};" >> ${SBUILD_CONFIG}
+    fi
+    export SBUILD_CONFIG="${SBUILD_CONFIG}"
+}
+
+insert_mounts() {
+    sudo -s <<'EOSUDO'
+        set -e
+        for mp in ${SCHROOT_MOUNTS}; do
+            FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
+            grep -qxF "${FSTAB_LINE}" ${SBUILD_CONF_DIR}/fstab || \
+                echo "${FSTAB_LINE}" >> ${SBUILD_CONF_DIR}/fstab
+        done
+EOSUDO
+}
+
+remove_mounts() {
+    sudo -s <<'EOSUDO'
+        set -e
+        for mp in ${SCHROOT_MOUNTS}; do
+            FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
+            sed -i "\|${FSTAB_LINE}|d" ${SBUILD_CONF_DIR}/fstab
+        done
+EOSUDO
+}
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 20a218f..6451cb5 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -58,6 +58,8 @@ SSTATE_DIR ?= "${TOPDIR}/sstate-cache"
 SSTATE_MANIFESTS = "${TMPDIR}/sstate-control/${DISTRO}-${DISTRO_ARCH}"
 BUILDCHROOT_HOST_DIR = "${DEPLOY_DIR_BUILDCHROOT}-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
 BUILDCHROOT_TARGET_DIR = "${DEPLOY_DIR_BUILDCHROOT}-target/${DISTRO}-${DISTRO_ARCH}"
+SCHROOT_HOST_DIR = "${DEPLOY_DIR}/schroot-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
+SCHROOT_TARGET_DIR = "${DEPLOY_DIR}/schroot-target/${DISTRO}-${DISTRO_ARCH}"
 SDKCHROOT_DIR = "${DEPLOY_DIR_SDKCHROOT}/${BPN}-${DISTRO}-${MACHINE}"
 CACHE = "${TMPDIR}/cache"
 KERNEL_FILE ?= "vmlinuz"
-- 
2.17.1

[PATCH v12 05/21] dpkg: Build packages with sbuild

[Anton Mikanovich]

Use previously exported to schroot rootfs to build the package with
sbuild tool. Provide isar-apt as extra repository to be automatically
add to apt sources.

Also added /home/.git-downloads mount while it's used
as git alternates location.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 31 ++++++++++++++++---------------
 meta/classes/dpkg-gbp.bbclass  |  2 ++
 meta/classes/dpkg.bbclass      | 10 ++++++++--
 meta/classes/sbuild.bbclass    |  4 ++--
 4 files changed, 28 insertions(+), 19 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 7e032ba..dbe35f4 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -4,6 +4,7 @@
 #
 # SPDX-License-Identifier: MIT
 
+inherit sbuild
 inherit buildchroot
 inherit debianize
 inherit terminal
@@ -16,6 +17,8 @@ DEPENDS_append_riscv64 = "${@' crossbuild-essential-riscv64' if d.getVar('ISAR_C
 DEB_BUILD_PROFILES ?= ""
 DEB_BUILD_OPTIONS ?= ""
 
+ISAR_APT_REPO ?= "deb [trusted=yes] file:///home/builder/${PN}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO} ${DEBDISTRONAME} main"
+
 python do_adjust_git() {
     import subprocess
 
@@ -175,6 +178,7 @@ addtask prepare_build after do_patch do_transform_template before do_dpkg_build
 # If Isar recipes depend on each other, they typically need the package
 # deployed to isar-apt
 do_prepare_build[deptask] = "do_deploy_deb"
+do_prepare_build[depends] = "${SCHROOT_DEP}"
 
 BUILDROOT = "${BUILDCHROOT_DIR}/${PP}"
 
@@ -200,6 +204,15 @@ dpkg_undo_mounts() {
     sudo rmdir ${BUILDROOT}
 }
 
+do_prepare_build_append() {
+    # Make a local copy of isar-apt repo that is not affected by other parallel builds
+    mkdir -p ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
+    rm -rf ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/*
+    cp -Rl ${REPO_ISAR_DIR} ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
+}
+
+do_prepare_build[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
+
 # Placeholder for actual dpkg_runbuild() implementation
 dpkg_runbuild() {
     die "This should never be called, overwrite it in your derived class"
@@ -222,14 +235,11 @@ def isar_export_build_settings(d):
     os.environ['DEB_BUILD_PROFILES'] = isar_deb_build_profiles(d)
 
 python do_dpkg_build() {
-    lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
-                             shared=True)
-    bb.build.exec_func("dpkg_do_mounts", d)
+    bb.build.exec_func('schroot_create_configs', d)
     try:
         bb.build.exec_func("dpkg_runbuild", d)
     finally:
-        bb.build.exec_func("dpkg_undo_mounts", d)
-        bb.utils.unlockfile(lock)
+        bb.build.exec_func('schroot_delete_configs', d)
 }
 
 addtask dpkg_build
@@ -259,7 +269,7 @@ python do_dpkg_build_setscene() {
 addtask dpkg_build_setscene
 do_dpkg_build_setscene[dirs] += "${S}/.."
 
-KEEP_INSTALLED_ON_CLEAN ?= "0"
+do_dpkg_build[depends] = "${SCHROOT_DEP}"
 
 CLEANFUNCS += "deb_clean"
 
@@ -269,15 +279,6 @@ deb_clean() {
         for d in ${DEBS}; do
             repo_del_package "${REPO_ISAR_DIR}"/"${DISTRO}" \
                 "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${d}"
-            if [ "${KEEP_INSTALLED_ON_CLEAN}" = "1" ]; then
-                continue;
-            fi
-            package=$(basename "${d}")
-            package_remove="/usr/bin/apt-get remove -y ${package%%_*}"
-            sudo -E chroot ${BUILDCHROOT_DIR} ${package_remove} || true
-            if [ "${BUILDCHROOT_DIR}" != "${BUILDCHROOT_TARGET_DIR}" ]; then
-                    sudo -E chroot ${BUILDCHROOT_TARGET_DIR} ${package_remove} || true
-            fi
         done
     fi
 }
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index 2b0b818..68172b3 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -11,6 +11,8 @@ PATCHTOOL ?= "git"
 
 GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
 
+SCHROOT_MOUNTS = "${WORKDIR}:${PP} ${GITDIR}:/home/.git-downloads"
+
 dpkg_runbuild_prepend() {
     sh -c "
         cd ${WORKDIR}/${PPS}
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index af83353..5f68fd6 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -39,6 +39,12 @@ dpkg_runbuild() {
     export DEB_BUILD_OPTIONS="${@ isar_deb_build_options(d)}"
     export DEB_BUILD_PROFILES="${@ isar_deb_build_profiles(d)}"
     export PARALLEL_MAKE="${PARALLEL_MAKE}"
-    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
-         /isar/build.sh ${PP}/${PPS} ${PACKAGE_ARCH}
+
+    export SBUILD_CONFIG="${SBUILD_CONFIG}"
+
+    sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
+        --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \
+        --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
+        --debbuildopts="--source-option=-I" \
+        --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
 }
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index b9fc6a0..a3783bc 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -29,6 +29,8 @@ SBUILD_CHROOT ?= "${DEBDISTRONAME}-${SCHROOT_USER}-${@os.getpid()}"
 SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}"
 SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}"
 
+SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
+
 schroot_create_configs() {
     sudo -s <<'EOSUDO'
         set -e
@@ -73,7 +75,6 @@ EOSUDO
 }
 
 sbuild_export() {
-    SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
     VAR=${1}; shift
     VAR_LINE="'${VAR}' => '${@}',"
     if [ -s "${SBUILD_CONFIG}" ]; then
@@ -83,7 +84,6 @@ sbuild_export() {
         echo "${VAR_LINE}" >> ${SBUILD_CONFIG}
         echo "};" >> ${SBUILD_CONFIG}
     fi
-    export SBUILD_CONFIG="${SBUILD_CONFIG}"
 }
 
 insert_mounts() {
-- 
2.17.1

[PATCH v12 06/21] sbuild: Support of DEB_BUILD_PROFILES

[Anton Mikanovich]

From: Uladzimir Bely <ubely@ilbers.de>

In Isar packages may use their own profiles. For example, linux-custom.inc
introduces 'nolibcdev' profile that should be passed to dpkg-buildpackage
in both cross and native builds.

By default, sbuild for cross-building uses passes "cross,nocheck" options
to dpkg-buildpackage and ignores DEB_BUILD_PROFILES environment variable.

This change makes sbuild use custom profiles even in cross-build mode.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 5f68fd6..218c7e8 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -37,13 +37,17 @@ dpkg_runbuild() {
     E="${@ isar_export_proxies(d)}"
     E="${@ isar_export_ccache(d)}"
     export DEB_BUILD_OPTIONS="${@ isar_deb_build_options(d)}"
-    export DEB_BUILD_PROFILES="${@ isar_deb_build_profiles(d)}"
     export PARALLEL_MAKE="${PARALLEL_MAKE}"
 
+    profiles="${@ isar_deb_build_profiles(d)}"
+    if [ ! -z "$profiles" ]; then
+        profiles=$(echo --profiles="$profiles" | sed -e 's/ \+/,/g')
+    fi
+
     export SBUILD_CONFIG="${SBUILD_CONFIG}"
 
     sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
-        --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \
+        --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
         --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
         --debbuildopts="--source-option=-I" \
         --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
-- 
2.17.1

[PATCH v12 07/21] sbuild: Support of shell exports from dpkg_runbuild_prepend

[Anton Mikanovich]

From: Uladzimir Bely <ubely@ilbers.de>

Many of recipes often use shell exports done in dpkg_run_prepend.

While sbuild is performed in isolated environment, we need a way to pass
these exported variables to it. This is done by storing environment just
before dpkg_runbuild (after prepare_build) and finding just before
the actual build what has been changed or added.

This patch allows using shell exports, but warning will be produced.
It's recommended to modify the code (in downstreams) and replace shell
exports by other methods, like using templates.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 218c7e8..05b46c9 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -32,6 +32,12 @@ do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
 
 addtask devshell after do_install_builddeps
 
+DPKG_PREBUILD_ENV_FILE="${WORKDIR}/dpkg_prebuild.env"
+
+do_prepare_build_append() {
+    env > ${DPKG_PREBUILD_ENV_FILE}
+}
+
 # Build package from sources using build script
 dpkg_runbuild() {
     E="${@ isar_export_proxies(d)}"
@@ -39,6 +45,26 @@ dpkg_runbuild() {
     export DEB_BUILD_OPTIONS="${@ isar_deb_build_options(d)}"
     export PARALLEL_MAKE="${PARALLEL_MAKE}"
 
+    env | while read -r line; do
+        # Filter the same lines
+        grep -q "^${line}" ${DPKG_PREBUILD_ENV_FILE} && continue
+        # Filter some standard variables
+        echo ${line} | grep -q "^HOME=" && continue
+        echo ${line} | grep -q "^PWD=" && continue
+
+        var=$(echo "${line}" | cut -d '=' -f1)
+        value=$(echo "${line}" | cut -d '=' -f2-)
+        sbuild_export $var "$value"
+
+        # Don't warn some variables
+        [ "${var}" = "PARALLEL_MAKE" ] && continue
+        [ "${var}" = "CCACHE_DIR" ] && continue
+        [ "${var}" = "PATH_PREPEND" ] && continue
+        [ "${var}" = "DEB_BUILD_OPTIONS" ] && continue
+
+        bbwarn "Export of '${line}' detected, please migrate to templates"
+    done
+
     profiles="${@ isar_deb_build_profiles(d)}"
     if [ ! -z "$profiles" ]; then
         profiles=$(echo --profiles="$profiles" | sed -e 's/ \+/,/g')
-- 
2.17.1

[PATCH v12 08/21] dpkg: Remove builddeps install task

[Anton Mikanovich]

From: Uladzimir Bely <ubely@ilbers.de>

While builddeps are now handled interlnally by sbuild, we don't need
a task to preinstall them in buildchroot.

But for the second local (base-apt) build we need a way to keep
all dependencies in form of .deb files in DL_DIR. This is done
by executing additional commands in sbuild that copy them to/from
an externally mounted folder.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/dpkg.bbclass   | 47 ++++++++++++++++---------------------
 meta/classes/sbuild.bbclass |  1 -
 2 files changed, 20 insertions(+), 28 deletions(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 05b46c9..3c7b670 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -5,33 +5,6 @@ inherit dpkg-base
 
 PACKAGE_ARCH ?= "${DISTRO_ARCH}"
 
-# Install build dependencies for package
-do_install_builddeps() {
-    dpkg_do_mounts
-    E="${@ isar_export_proxies(d)}"
-    export DEB_BUILD_OPTIONS="${@ isar_deb_build_options(d)}"
-    export DEB_BUILD_PROFILES="${@ isar_deb_build_profiles(d)}"
-    distro="${DISTRO}"
-    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
-        distro="${HOST_DISTRO}"
-    fi
-
-    deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
-        ${PP}/${PPS} ${PACKAGE_ARCH} --download-only
-    deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
-        ${PP}/${PPS} ${PACKAGE_ARCH}
-    dpkg_undo_mounts
-}
-
-addtask install_builddeps after do_prepare_build before do_dpkg_build
-do_install_builddeps[depends] += "${BUILDCHROOT_DEP} isar-apt:do_cache_config"
-# apt and reprepro may not run in parallel, acquire the Isar lock
-do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
-
-addtask devshell after do_install_builddeps
-
 DPKG_PREBUILD_ENV_FILE="${WORKDIR}/dpkg_prebuild.env"
 
 do_prepare_build_append() {
@@ -65,6 +38,16 @@ dpkg_runbuild() {
         bbwarn "Export of '${line}' detected, please migrate to templates"
     done
 
+    distro="${DISTRO}"
+    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+        distro="${HOST_DISTRO}"
+    fi
+
+    deb_dl_dir_import "${WORKDIR}/rootfs" "${distro}"
+
+    deb_dir="/var/cache/apt/archives"
+    ext_deb_dir="${PP}/rootfs/${deb_dir}"
+
     profiles="${@ isar_deb_build_profiles(d)}"
     if [ ! -z "$profiles" ]; then
         profiles=$(echo --profiles="$profiles" | sed -e 's/ \+/,/g')
@@ -72,9 +55,19 @@ dpkg_runbuild() {
 
     export SBUILD_CONFIG="${SBUILD_CONFIG}"
 
+    echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
+
     sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
         --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
         --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
+        --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
+        --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
+        --finished-build-commands="cp -n --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
         --debbuildopts="--source-option=-I" \
         --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
+
+    deb_dl_dir_export "${WORKDIR}/rootfs" "${distro}"
+
+    # Cleanup apt artifacts
+    sudo rm -rf ${WORKDIR}/rootfs
 }
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index a3783bc..178ba1b 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -8,7 +8,6 @@ SCHROOT_MOUNTS ?= ""
 python __anonymous() {
     import pwd
     d.setVar('SCHROOT_USER', pwd.getpwuid(os.geteuid()).pw_name)
-    d.setVar('SCHROOT_USER_HOME', pwd.getpwuid(os.geteuid()).pw_dir)
 
     mode = d.getVar('ISAR_CROSS_COMPILE', True)
     distro_arch = d.getVar('DISTRO_ARCH')
-- 
2.17.1

[PATCH v12 09/21] sbuild: Add ccache support

[Anton Mikanovich]

From: Uladzimir Bely <ubely@ilbers.de>

This adds ccache support for custom packages in the same manner
as it was done previously for buildchroot-based build, by using
USE_CCACHE variable in local.conf or per-recipe.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass                     |  4 ++++
 meta/classes/sbuild.bbclass                   | 23 +++++++++++++++++++
 .../sbuild-chroot/sbuild-chroot.inc           |  1 +
 3 files changed, 28 insertions(+)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 3c7b670..d92135c 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -48,6 +48,10 @@ dpkg_runbuild() {
     deb_dir="/var/cache/apt/archives"
     ext_deb_dir="${PP}/rootfs/${deb_dir}"
 
+    if [ ${USE_CCACHE} -eq 1 ]; then
+        schroot_configure_ccache
+    fi
+
     profiles="${@ isar_deb_build_profiles(d)}"
     if [ ! -z "$profiles" ]; then
         profiles=$(echo --profiles="$profiles" | sed -e 's/ \+/,/g')
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 178ba1b..6ff41dd 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -105,3 +105,26 @@ remove_mounts() {
         done
 EOSUDO
 }
+
+schroot_configure_ccache() {
+    sudo -s <<'EOSUDO'
+        set -e
+
+        sbuild_fstab="${SBUILD_CONF_DIR}/fstab"
+
+        install --group=sbuild --mode=2775 -d ${CCACHE_DIR}
+        fstab_ccachedir="${CCACHE_DIR} /ccache none rw,bind 0 0"
+        grep -qxF "${fstab_ccachedir}" ${sbuild_fstab} || echo "${fstab_ccachedir}" >> ${sbuild_fstab}
+
+        (flock 9
+        [ -w ${CCACHE_DIR}/sbuild-setup ] || cat << END > ${CCACHE_DIR}/sbuild-setup
+#!/bin/sh
+export PATH="\$PATH_PREPEND:\$PATH"
+exec "\$@"
+END
+        chmod a+rx ${CCACHE_DIR}/sbuild-setup
+        ) 9>"${CCACHE_DIR}/sbuild-setup.lock"
+
+        echo "command-prefix=/ccache/sbuild-setup" >> "${SCHROOT_CONF_FILE}"
+EOSUDO
+}
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
index 177a8a6..74811e2 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -14,6 +14,7 @@ SBUILD_CHROOT_PREINSTALL_COMMON = " \
     fakeroot \
     build-essential \
     debhelper \
+    ccache \
 "
 
 SBUILD_CHROOT_PREINSTALL ?= " \
-- 
2.17.1

[PATCH v12 10/21] dpkg-base: Switch devshell to use schroot

[Anton Mikanovich]

From: Uladzimir Bely <ubely@ilbers.de>

While packages are now build in schroot with sbuild, devshell
task should make terminal open inside sbuild environment.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-base.bbclass                | 23 ++++++++++++++-----
 meta/classes/sbuild.bbclass                   |  3 +++
 .../sbuild-chroot/sbuild-chroot.inc           |  2 ++
 3 files changed, 22 insertions(+), 6 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index dbe35f4..7005aa5 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -298,19 +298,30 @@ do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
 do_deploy_deb[dirs] = "${S}"
 
 python do_devshell() {
-    bb.build.exec_func('dpkg_do_mounts', d)
+    bb.build.exec_func('schroot_create_configs', d)
 
     isar_export_proxies(d)
     isar_export_ccache(d)
     isar_export_build_settings(d)
 
-    buildchroot = d.getVar('BUILDCHROOT_DIR')
+    schroot = d.getVar('SBUILD_CHROOT')
+    isar_apt = d.getVar('ISAR_APT_REPO')
     pp_pps = os.path.join(d.getVar('PP'), d.getVar('PPS'))
-    # the PATH variable is not forwarded by sudo -E.
-    termcmd = "sudo -E chroot {0} sh -c 'cd {1}; export PATH=$PATH_PREPEND:$PATH; $SHELL -i'"
-    oe_terminal(termcmd.format(buildchroot, pp_pps), "Isar devshell", d)
 
-    bb.build.exec_func('dpkg_undo_mounts', d)
+    install_deps = ":" if d.getVar('BB_CURRENTTASK') == "devshell_nodeps" else "mk-build-deps -i -t \
+        \"apt-get -y -q -o Debug::pkgProblemResolver=yes --no-install-recommends --allow-downgrades\" \
+        debian/control"
+
+    termcmd = "schroot -d / -c {0} -u root -- sh -c ' \
+        cd {1}; \
+        echo {2} > /etc/apt/sources.list.d/isar_apt.list; \
+        apt-get -y -q update; \
+        {3}; \
+        $SHELL -i \
+    '"
+    oe_terminal(termcmd.format(schroot, pp_pps, isar_apt, install_deps), "Isar devshell", d)
+
+    bb.build.exec_func('schroot_delete_configs', d)
 }
 
 addtask devshell after do_prepare_build
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 6ff41dd..2d52307 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -56,6 +56,9 @@ EOF
         fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind 0 0"
         grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}
 
+        fstab_pkgdir="${WORKDIR} /home/builder/${PN} none rw,bind 0 0"
+        grep -qxF "${fstab_pkgdir}" ${sbuild_fstab} || echo "${fstab_pkgdir}" >> ${sbuild_fstab}
+
         if [ -d ${DL_DIR} ]; then
             fstab_downloads="${DL_DIR} /downloads none rw,bind 0 0"
             grep -qxF "${fstab_downloads}" ${sbuild_fstab} || echo "${fstab_downloads}" >> ${sbuild_fstab}
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
index 74811e2..4688da1 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -15,6 +15,8 @@ SBUILD_CHROOT_PREINSTALL_COMMON = " \
     build-essential \
     debhelper \
     ccache \
+    devscripts \
+    equivs \
 "
 
 SBUILD_CHROOT_PREINSTALL ?= " \
-- 
2.17.1

[PATCH v12 11/21] dpkg-base: Switch apt_fetch and apt_unpack to use schroot

[Anton Mikanovich]

From: Uladzimir Bely <ubely@ilbers.de>

This moves downloading and unpacking deb-src to schroot environment.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 25 +++++++++----------------
 1 file changed, 9 insertions(+), 16 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 7005aa5..87dc3f7 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -104,19 +104,12 @@ python() {
 }
 
 do_apt_fetch() {
-    dpkg_do_mounts
-    E="${@ isar_export_proxies(d)}"
-    sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update \
-        -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
-        -o Dir::Etc::SourceParts="-" \
-        -o APT::Get::List-Cleanup="0"
-
+    schroot_create_configs
     for uri in "${SRC_APT}"; do
-        sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+        schroot -d / -c ${SBUILD_CHROOT} -- \
             sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only-source source "$2"' my_script "${DISTRO}" "${uri}"
     done
-
-    dpkg_undo_mounts
+    schroot_delete_configs
 }
 
 addtask apt_fetch
@@ -125,13 +118,14 @@ do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
 # Add dependency from the correct buildchroot: host or target
 do_apt_fetch[depends] += "${BUILDCHROOT_DEP}"
 
+# Add dependency from the correct schroot: host or target
+do_apt_fetch[depends] += "${SCHROOT_DEP}"
+
 do_apt_unpack() {
     rm -rf ${S}
-    dpkg_do_mounts
-    E="${@ isar_export_proxies(d)}"
-
+    schroot_create_configs
     for uri in "${SRC_APT}"; do
-        sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+        schroot -d / -c ${SBUILD_CHROOT} -- \
             sh -c ' \
                 set -e
                 dscfile="$(apt-get -y -qq --print-uris --only-source source "${2}" | cut -d " " -f2 | grep -E "*.dsc")"
@@ -140,8 +134,7 @@ do_apt_unpack() {
                 dpkg-source -x "${dscfile}" "${PPS}"' \
                     my_script "${DISTRO}" "${uri}"
     done
-
-    dpkg_undo_mounts
+    schroot_delete_configs
 }
 
 addtask apt_unpack after do_apt_fetch
-- 
2.17.1

[PATCH v12 12/21] doc: Add sbuild-related documentation

[Anton Mikanovich]

From: Uladzimir Bely <ubely@ilbers.de>

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 doc/user_manual.md | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 61834af..3faa8f1 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -86,6 +86,24 @@ apt install \
   pristine-tar
 ```
 
+Additional setup is required since `sbuild` is now used for package build.
+Install the following packages:
+```
+apt install \
+  sbuild \
+  schroot
+```
+Also, user who runs isar should be added to `sbuild` group.
+
+**NOTE:** sbuild version (<=0.78.1) packaged in Debian Buster doesn't support
+`$apt_keep_downloaded_packages` option which is required in Isar for
+populating `${DL_DIR}/deb`. So, host `sbuild` in this case should be manually
+upgraded to >=0.81.2 version from Debian Bullseye.
+
+```
+sudo gpasswd -a <username> sbuild
+```
+
 If your host is >= buster, also install the following package.
 ```
 apt install python3-distutils
-- 
2.17.1

[PATCH v12 13/21] sbuild: Use .dsc file instead of source directory

[Anton Mikanovich]

From: Uladzimir Bely <ubely@ilbers.de>

This makes build more Debian-like and avoids additional execution of
"dh clean" by sbuild before dependencies resolved and build started.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index d92135c..564e654 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -61,6 +61,10 @@ dpkg_runbuild() {
 
     echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
 
+    # Create a .dsc file from source directory to use it with sbuild
+    sh -c "cd ${WORKDIR}; dpkg-source -q -b ${PPS}"
+    DSC=$(head -n1 ${WORKDIR}/${PPS}/debian/changelog | awk '{gsub(/[()]/,""); printf "%s_%s.dsc", $1, $2}')
+
     sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
         --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
         --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
@@ -68,7 +72,7 @@ dpkg_runbuild() {
         --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
         --finished-build-commands="cp -n --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
         --debbuildopts="--source-option=-I" \
-        --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
+        --build-dir=${WORKDIR} --dist="isar" ${WORKDIR}/${DSC}
 
     deb_dl_dir_export "${WORKDIR}/rootfs" "${distro}"
 
-- 
2.17.1

[PATCH v12 14/21] sbuild: Fixed proxy support

[Anton Mikanovich]

From: Uladzimir Bely <ubely@ilbers.de>

This makes sbuild able to work via proxy.

Proxy settings passed via 'sbuild_export' seem to work only for build
stage in sbuild. Previous stages (like schroot setup, downloading deps)
don't use this values.

To work via proxy in this stages there should be environment filter
for sbuild set via sbuild.conf.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-base.bbclass |  1 +
 meta/classes/dpkg.bbclass      | 14 ++++++++++++++
 meta/classes/sbuild.bbclass    | 31 ++++++++++++++++++++++++-------
 3 files changed, 39 insertions(+), 7 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 87dc3f7..7948b23 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -104,6 +104,7 @@ python() {
 }
 
 do_apt_fetch() {
+    E="${@ isar_export_proxies(d)}"
     schroot_create_configs
     for uri in "${SRC_APT}"; do
         schroot -d / -c ${SBUILD_CHROOT} -- \
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 564e654..d8c4dd8 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -35,6 +35,15 @@ dpkg_runbuild() {
         [ "${var}" = "PATH_PREPEND" ] && continue
         [ "${var}" = "DEB_BUILD_OPTIONS" ] && continue
 
+        [ "${var}" = "http_proxy" ] && continue
+        [ "${var}" = "HTTP_PROXY" ] && continue
+        [ "${var}" = "https_proxy" ] && continue
+        [ "${var}" = "HTTPS_PROXY" ] && continue
+        [ "${var}" = "ftp_proxy" ] && continue
+        [ "${var}" = "FTP_PROXY" ] && continue
+        [ "${var}" = "no_proxy" ] && continue
+        [ "${var}" = "NO_PROXY" ] && continue
+
         bbwarn "Export of '${line}' detected, please migrate to templates"
     done
 
@@ -59,6 +68,11 @@ dpkg_runbuild() {
 
     export SBUILD_CONFIG="${SBUILD_CONFIG}"
 
+    for envvar in http_proxy HTTP_PROXY https_proxy HTTPS_PROXY \
+        ftp_proxy FTP_PROXY no_proxy NO_PROXY; do
+        sbuild_add_env_filter "$envvar"
+    done
+
     echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
 
     # Create a .dsc file from source directory to use it with sbuild
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 2d52307..7d24c47 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -76,16 +76,33 @@ schroot_delete_configs() {
 EOSUDO
 }
 
+sbuild_add_env_filter() {
+    [ -w ${SBUILD_CONFIG} ] || touch ${SBUILD_CONFIG}
+
+    if ! grep -q "^\$environment_filter =" ${SBUILD_CONFIG}; then
+        echo "\$environment_filter = [" >> ${SBUILD_CONFIG}
+        echo "];" >> ${SBUILD_CONFIG}
+    fi
+
+    FILTER=${1}
+
+    sed -i -e "/'\^${FILTER}\\$/d" \
+        -e "/^\$environment_filter =.*/a '^${FILTER}\$'," ${SBUILD_CONFIG}
+}
+
 sbuild_export() {
-    VAR=${1}; shift
-    VAR_LINE="'${VAR}' => '${@}',"
-    if [ -s "${SBUILD_CONFIG}" ]; then
-        sed -i -e "\$i\\" -e "${VAR_LINE}" ${SBUILD_CONFIG}
-    else
-        echo "\$build_environment = {" > ${SBUILD_CONFIG}
-        echo "${VAR_LINE}" >> ${SBUILD_CONFIG}
+    [ -w ${SBUILD_CONFIG} ] || touch ${SBUILD_CONFIG}
+
+    if ! grep -q "^\$build_environment =" ${SBUILD_CONFIG}; then
+        echo "\$build_environment = {" >> ${SBUILD_CONFIG}
         echo "};" >> ${SBUILD_CONFIG}
     fi
+
+    VAR=${1}; shift
+    VAR_LINE="'${VAR}' => '${@}',"
+
+    sed -i -e "/^'${VAR}' =>/d" ${SBUILD_CONFIG} \
+        -e "/^\$build_environment =.*/a ${VAR_LINE}" ${SBUILD_CONFIG}
 }
 
 insert_mounts() {
-- 
2.17.1

[PATCH v12 15/21] sbuild: Fix debsrc_download for packages dependencies

[Anton Mikanovich]

From: Uladzimir Bely <ubely@ilbers.de>

With sbuild, build deps are internally installed in temporary sbuild
environment,so they don't appear in any external dpkg.log files.

With "cache-deb-src" base repo feature enabled, these such packages
are considered not to belong to current build, so their src packages
are not downloaded at image postprocess stage.

The patch fixes this by extracting dpkg.log files from the sbuild
environment to one common log file. This consolidated log is then
used at rootfs_postprocess to get source packages.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/deb-dl-dir.bbclass | 4 ++++
 meta/classes/dpkg.bbclass       | 6 +++++-
 meta/classes/sbuild.bbclass     | 9 +++++++++
 3 files changed, 18 insertions(+), 1 deletion(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 4780228..f479b29 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -14,6 +14,10 @@ is_not_part_of_current_build() {
     # build using the below method.
     local output="$( grep -xhs ".* status installed ${package}:${arch} ${version}" \
             "${IMAGE_ROOTFS}"/var/log/dpkg.log \
+            "${SCHROOT_HOST_DIR}"/var/log/dpkg.log \
+            "${SCHROOT_TARGET_DIR}"/var/log/dpkg.log \
+            "${SCHROOT_HOST_DIR}"/tmp/dpkg_common.log \
+            "${SCHROOT_TARGET_DIR}"/tmp/dpkg_common.log \
             "${BUILDCHROOT_HOST_DIR}"/var/log/dpkg.log \
             "${BUILDCHROOT_TARGET_DIR}"/var/log/dpkg.log | head -1 )"
 
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index d8c4dd8..a0544d5 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -55,7 +55,8 @@ dpkg_runbuild() {
     deb_dl_dir_import "${WORKDIR}/rootfs" "${distro}"
 
     deb_dir="/var/cache/apt/archives"
-    ext_deb_dir="${PP}/rootfs/${deb_dir}"
+    ext_root="${PP}/rootfs"
+    ext_deb_dir="${ext_root}${deb_dir}"
 
     if [ ${USE_CCACHE} -eq 1 ]; then
         schroot_configure_ccache
@@ -82,12 +83,15 @@ dpkg_runbuild() {
     sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
         --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
         --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
+        --chroot-setup-commands="rm -f /var/log/dpkg.log" \
         --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
         --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
         --finished-build-commands="cp -n --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
+        --finished-build-commands="cp /var/log/dpkg.log ${ext_root}/dpkg_partial.log" \
         --debbuildopts="--source-option=-I" \
         --build-dir=${WORKDIR} --dist="isar" ${WORKDIR}/${DSC}
 
+    sbuild_dpkg_log_export "${WORKDIR}/rootfs/dpkg_partial.log"
     deb_dl_dir_export "${WORKDIR}/rootfs" "${distro}"
 
     # Cleanup apt artifacts
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 7d24c47..a78b15b 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -148,3 +148,12 @@ END
         echo "command-prefix=/ccache/sbuild-setup" >> "${SCHROOT_CONF_FILE}"
 EOSUDO
 }
+
+sbuild_dpkg_log_export() {
+    export dpkg_partial_log="${1}"
+
+    ( flock 9
+    set -e
+    cat ${dpkg_partial_log} >> ${SCHROOT_DIR}/tmp/dpkg_common.log
+    )  9>"${SCHROOT_DIR}/tmp/dpkg_common.log.lock"
+}
-- 
2.17.1

[PATCH v12 16/21] fix: support build of packages with epoch version

[Anton Mikanovich]

From: Felix Moessbauer <felix.moessbauer@siemens.com>

This patch reworks the logic how the DSC file for building
with sbuild is located.
By that, we do not rely on implementation details of dpkg-source
regarding the name of the generated .dsc file.
Instead, we remove all <package>*.dsc files in advance, let dpkg-source
generate the file and use a find to locate the generated one.

By that, package versions with epochs (e.g. "1:1.0.0") are
supported.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/dpkg.bbclass | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index a0544d5..3dda7d2 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -77,8 +77,10 @@ dpkg_runbuild() {
     echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
 
     # Create a .dsc file from source directory to use it with sbuild
+    DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
+    find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -delete
     sh -c "cd ${WORKDIR}; dpkg-source -q -b ${PPS}"
-    DSC=$(head -n1 ${WORKDIR}/${PPS}/debian/changelog | awk '{gsub(/[()]/,""); printf "%s_%s.dsc", $1, $2}')
+    DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -print)
 
     sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
         --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
@@ -89,7 +91,7 @@ dpkg_runbuild() {
         --finished-build-commands="cp -n --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
         --finished-build-commands="cp /var/log/dpkg.log ${ext_root}/dpkg_partial.log" \
         --debbuildopts="--source-option=-I" \
-        --build-dir=${WORKDIR} --dist="isar" ${WORKDIR}/${DSC}
+        --build-dir=${WORKDIR} --dist="isar" ${DSC_FILE}
 
     sbuild_dpkg_log_export "${WORKDIR}/rootfs/dpkg_partial.log"
     deb_dl_dir_export "${WORKDIR}/rootfs" "${distro}"
-- 
2.17.1

[PATCH v12 17/21] always create apt-cache dirs in deb_dl_dir_import

[Anton Mikanovich]

From: Felix Moessbauer <felix.moessbauer@siemens.com>

This patch ensures that the deb_dl_dir_import function
always creates the apt-cache directories under ${PP}/rootfs,
no matter if files are in the download cache or not.

By that, downstream tasks can safely assume that the apt-cache
is created and place files there.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/deb-dl-dir.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index f479b29..188cb83 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -78,8 +78,8 @@ debsrc_download() {
 deb_dl_dir_import() {
     export pc="${DEBDIR}/${2}"
     export rootfs="${1}"
-    [ ! -d "${pc}" ] && return 0
     sudo mkdir -p "${rootfs}"/var/cache/apt/archives/
+    [ ! -d "${pc}" ] && return 0
     flock -s "${pc}".lock -c '
         set -e
         printenv | grep -q BB_VERBOSE_LOGS && set -x
-- 
2.17.1

[PATCH v12 18/21] avoid absolute SCHROOT_* paths to improve caching

[Anton Mikanovich]

From: Felix Moessbauer <felix.moessbauer@siemens.com>

This patch reworks the logic to setup the SCHROOT_* bitbake
variables by making sure nothing is expanded.
By that, no absolute paths into the build host are exposed,
which finally makes the recipe cachable.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/sbuild.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index a78b15b..f9bfaaf 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -14,11 +14,11 @@ python __anonymous() {
     if mode == "0" or d.getVar('HOST_ARCH') ==  distro_arch or \
        (d.getVar('HOST_DISTRO') == "debian-stretch" and distro_arch == "i386"):
         d.setVar('SBUILD_HOST_ARCH', distro_arch)
-        d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_TARGET_DIR'))
+        d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_TARGET_DIR', False))
         dep = "sbuild-chroot-target:do_build"
     else:
         d.setVar('SBUILD_HOST_ARCH', d.getVar('HOST_ARCH'))
-        d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_HOST_DIR'))
+        d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_HOST_DIR', False))
         dep = "sbuild-chroot-host:do_build"
     d.setVar('SCHROOT_DEP', dep)
 }
-- 
2.17.1

[PATCH v12 19/21] dpkg-base: Cleanup on schroot fail

[Anton Mikanovich]

Cleanup schroot configs in case schroot exit with non zero state.
Handle all possible interrupt cases with trap.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 7948b23..39ab04f 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -106,6 +106,13 @@ python() {
 do_apt_fetch() {
     E="${@ isar_export_proxies(d)}"
     schroot_create_configs
+
+    schroot_cleanup() {
+        schroot_delete_configs
+    }
+    trap 'exit 1' INT HUP QUIT TERM ALRM USR1
+    trap 'schroot_cleanup' EXIT
+
     for uri in "${SRC_APT}"; do
         schroot -d / -c ${SBUILD_CHROOT} -- \
             sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only-source source "$2"' my_script "${DISTRO}" "${uri}"
@@ -125,6 +132,13 @@ do_apt_fetch[depends] += "${SCHROOT_DEP}"
 do_apt_unpack() {
     rm -rf ${S}
     schroot_create_configs
+
+    schroot_cleanup() {
+        schroot_delete_configs
+    }
+    trap 'exit 1' INT HUP QUIT TERM ALRM USR1
+    trap 'schroot_cleanup' EXIT
+
     for uri in "${SRC_APT}"; do
         schroot -d / -c ${SBUILD_CHROOT} -- \
             sh -c ' \
-- 
2.17.1

[PATCH v12 20/21] locate generated binary package in WORKDIR

[Anton Mikanovich]

From: Felix Moessbauer <felix.moessbauer@siemens.com>

With sbuilder, the generated binary packages are always
emitted into the WORKDIR indepedent of the value of ${S}.

Prior to this fix, this only worked if ${S} was a direct
subfolder of ${WORKDIR} (but no nested folder).

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/dpkg-base.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 39ab04f..79c1ce1 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -296,7 +296,7 @@ do_clean[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
 do_deploy_deb() {
     deb_clean
     repo_add_packages "${REPO_ISAR_DIR}"/"${DISTRO}" \
-        "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" ${S}/../*.deb
+        "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" ${WORKDIR}/*.deb
 }
 
 addtask deploy_deb after do_dpkg_build before do_build
-- 
2.17.1

[PATCH v12 21/21] sbuild: Add changelog entry

[Anton Mikanovich]

Migration from buildchroot to sbuild will cause API change.
Add changelog entry which also includes do_install_builddeps task
remove mentioning.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 RECIPE-API-CHANGELOG.md | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index cd6e756..621d110 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -406,3 +406,16 @@ groups are now created after all packages are installed.
 
 Changes in next
 ---------------
+
+### Buildchroot no longer used for package building
+
+Packages are now built with sbuild which takes care of dependency
+installation.
+The task do_install_builddeps has been removed.
+
+The migration to sbuild also means that all changes in the rootfs made during
+package building will not be shared between the build sessions of different
+packages and will be lost after a given build session finishes.
+
+Any package build requirements for the rootfs should be satisfied in the
+Debian way via package dependencies.
-- 
2.17.1

Re: [PATCH v12 00/21] Sbuild/Schroot migration

[Anton Mikanovich]

28.06.2022 09:48, Anton Mikanovich wrote:
> This patchset integrates sbuild/schroot tools into Isar build system.
>
> Base schroot images are created with sbuild-chroot-target or, in case
> of cross-build, sbuild-chroot-host recipes. These images are based on
> isar-bootstrap ones, but include some build-related stuff preinstalled.
>
> To use these changes you need to have sbuild and schroot installed and
> configured. User who runs build should be added to sbuild group.
>
> Currently, sbuild-related stuff is supported in latest kas.
>
> If Gilab CI is used, the following configuration changes are required:
> - external dir for schroot overlay should be set in the config
> /etc/gitlab-runner/config.toml:
>
> volumes = ["/path/to/overlay:/var/lib/schroot/union/overlay"]
>
> The patchset was tested with 'meta-iot2050' and 'xenomai-images'
> downstreams in combination with kas image mentioned above.
>
> Current limitations:
> - parallel building of several packages requires more free space in
> comparison with buildchroot-based architecture. This happens due
> the sbuild architecture which uses some basic layer (common for all
> packages) and per-package separate layer (where builddeps are downloaded
> and installed).
>
> Changes sinse v11:
> - Rebase on current next.
>
> Changes sinse v10:
> - Rebase on current next.
>
> Changes sinse v9:
> - Add caching improvement patch;
> - Remove per-package deb artifacts dir after usage to reduce disk space;
> - Improve schroot cleanup on failures.
>
> Changes since v8:
> - Added patch with support of package versions with epochs
>
> Changes since v7:
> - fixed work with proxy by adding some environment filters in sbuild;
> - DEB_BUILD_PROFILES support simplified and improved
> - Fixed gbp-based recipes rebuilding (get back --git-ignore-new option
> previously removed)
> - fixed base-apt caching for packages and source packages temporary
> installed in sbuild isolated environment
> - fixed rare build error related to ccache setup (added locking)
>
> Changes since v6:
> - added patch that switches sbuild from using source directory to using
> .dsc file. The file is generated before sbuild call by `dpkg-source`.
>
> Changes since v5:
> - warning about shell exports in dpkg_runbuild_prepend. This should
> avoided be reworked to templates.
>
> Changes since v4:
> - consider shell exports done in dpkg_runbuild_prepend and pass them
> to sbuild environment;
> - fixed ccache work.
>
> Changes since v3:
> - dpkg_do_mounts() and dpkg_undo_mounts() are not removed for downstreams
> compatibility;
> - dpkg_build_export is used for adjusting sbuild environment;
> - DEB_BUILD_PROFILES support for sbuild is fixed in case of cross-build.
>
> Changes since v2:
> - patches reworked/squashed for easier reading and understanding;
> - fixed building foreigh architectures with kas-docker;
> - implemented support of ccache;
> - fixed devshell and devshell_nodeps.
>
> Changes since v1:
> - parallel builds with different BUILD_DIR are supported;
> - parallel multiconfig targets in one build are supported;
> - per-task schroot configuration in /etc/schroot/ is now used;
> - patchset now passes Jenkins CI (so patches changes RFC => PATCH).
>
> Anton Mikanovich (3):
>    dpkg: Build packages with sbuild
>    dpkg-base: Cleanup on schroot fail
>    sbuild: Add changelog entry
>
> Felix Moessbauer (4):
>    fix: support build of packages with epoch version
>    always create apt-cache dirs in deb_dl_dir_import
>    avoid absolute SCHROOT_* paths to improve caching
>    locate generated binary package in WORKDIR
>
> Uladzimir Bely (14):
>    dpkg-gbp: Use separate command to export tarball
>    dpkg-gbp: Use host tools for dsc preparation
>    sbuild: Add recipes for host and target rootfs to run sbuild
>    sbuild: Introduce a class for another build method
>    sbuild: Support of DEB_BUILD_PROFILES
>    sbuild: Support of shell exports from dpkg_runbuild_prepend
>    dpkg: Remove builddeps install task
>    sbuild: Add ccache support
>    dpkg-base: Switch devshell to use schroot
>    dpkg-base: Switch apt_fetch and apt_unpack to use schroot
>    doc: Add sbuild-related documentation
>    sbuild: Use .dsc file instead of source directory
>    sbuild: Fixed proxy support
>    sbuild: Fix debsrc_download for packages dependencies
>
>   RECIPE-API-CHANGELOG.md                       |  13 ++
>   doc/user_manual.md                            |  22 ++-
>   meta/classes/deb-dl-dir.bbclass               |   6 +-
>   meta/classes/dpkg-base.bbclass                |  90 ++++++----
>   meta/classes/dpkg-gbp.bbclass                 |  26 +--
>   meta/classes/dpkg.bbclass                     | 109 +++++++++---
>   meta/classes/sbuild.bbclass                   | 159 ++++++++++++++++++
>   meta/conf/bitbake.conf                        |   2 +
>   .../sbuild-chroot/sbuild-chroot-host.bb       |  13 ++
>   .../sbuild-chroot/sbuild-chroot-target.bb     |  10 ++
>   .../sbuild-chroot/sbuild-chroot.inc           |  39 +++++
>   11 files changed, 408 insertions(+), 81 deletions(-)
>   create mode 100644 meta/classes/sbuild.bbclass
>   create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
>   create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
>   create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
>
This is also just a rebase on the current next.
We are planning to merge it in the next few days.

Bitbake update will be the first priority after that.

RE: [PATCH v12 00/21] Sbuild/Schroot migration

[Moessbauer, Felix]

> -----Original Message-----
> From: isar-users@googlegroups.com <isar-users@googlegroups.com> On
> Behalf Of Anton Mikanovich
> Sent: Tuesday, June 28, 2022 8:57 AM
> To: isar-users <isar-users@googlegroups.com>
> Subject: Re: [PATCH v12 00/21] Sbuild/Schroot migration
> 
> 28.06.2022 09:48, Anton Mikanovich wrote:
> > This patchset integrates sbuild/schroot tools into Isar build system.
> >
> > Base schroot images are created with sbuild-chroot-target or, in case
> > of cross-build, sbuild-chroot-host recipes. These images are based on
> > isar-bootstrap ones, but include some build-related stuff preinstalled.
> >
> > To use these changes you need to have sbuild and schroot installed and
> > configured. User who runs build should be added to sbuild group.
> >
> > Currently, sbuild-related stuff is supported in latest kas.
> >
> > If Gilab CI is used, the following configuration changes are required:
> > - external dir for schroot overlay should be set in the config
> > /etc/gitlab-runner/config.toml:
> >
> > volumes = ["/path/to/overlay:/var/lib/schroot/union/overlay"]
> >
> > The patchset was tested with 'meta-iot2050' and 'xenomai-images'
> > downstreams in combination with kas image mentioned above.
> >
> > Current limitations:
> > - parallel building of several packages requires more free space in
> > comparison with buildchroot-based architecture. This happens due the
> > sbuild architecture which uses some basic layer (common for all
> > packages) and per-package separate layer (where builddeps are
> > downloaded and installed).
> >
> > Changes sinse v11:
> > - Rebase on current next.
> >
> > Changes sinse v10:
> > - Rebase on current next.
> >
> > Changes sinse v9:
> > - Add caching improvement patch;
> > - Remove per-package deb artifacts dir after usage to reduce disk
> > space;
> > - Improve schroot cleanup on failures.
> >
> > Changes since v8:
> > - Added patch with support of package versions with epochs
> >
> > Changes since v7:
> > - fixed work with proxy by adding some environment filters in sbuild;
> > - DEB_BUILD_PROFILES support simplified and improved
> > - Fixed gbp-based recipes rebuilding (get back --git-ignore-new option
> > previously removed)
> > - fixed base-apt caching for packages and source packages temporary
> > installed in sbuild isolated environment
> > - fixed rare build error related to ccache setup (added locking)
> >
> > Changes since v6:
> > - added patch that switches sbuild from using source directory to
> > using .dsc file. The file is generated before sbuild call by `dpkg-source`.
> >
> > Changes since v5:
> > - warning about shell exports in dpkg_runbuild_prepend. This should
> > avoided be reworked to templates.
> >
> > Changes since v4:
> > - consider shell exports done in dpkg_runbuild_prepend and pass them
> > to sbuild environment;
> > - fixed ccache work.
> >
> > Changes since v3:
> > - dpkg_do_mounts() and dpkg_undo_mounts() are not removed for
> > downstreams compatibility;
> > - dpkg_build_export is used for adjusting sbuild environment;
> > - DEB_BUILD_PROFILES support for sbuild is fixed in case of cross-build.
> >
> > Changes since v2:
> > - patches reworked/squashed for easier reading and understanding;
> > - fixed building foreigh architectures with kas-docker;
> > - implemented support of ccache;
> > - fixed devshell and devshell_nodeps.
> >
> > Changes since v1:
> > - parallel builds with different BUILD_DIR are supported;
> > - parallel multiconfig targets in one build are supported;
> > - per-task schroot configuration in /etc/schroot/ is now used;
> > - patchset now passes Jenkins CI (so patches changes RFC => PATCH).
> >
> > Anton Mikanovich (3):
> >    dpkg: Build packages with sbuild
> >    dpkg-base: Cleanup on schroot fail
> >    sbuild: Add changelog entry
> >
> > Felix Moessbauer (4):
> >    fix: support build of packages with epoch version
> >    always create apt-cache dirs in deb_dl_dir_import
> >    avoid absolute SCHROOT_* paths to improve caching
> >    locate generated binary package in WORKDIR
> >
> > Uladzimir Bely (14):
> >    dpkg-gbp: Use separate command to export tarball
> >    dpkg-gbp: Use host tools for dsc preparation
> >    sbuild: Add recipes for host and target rootfs to run sbuild
> >    sbuild: Introduce a class for another build method
> >    sbuild: Support of DEB_BUILD_PROFILES
> >    sbuild: Support of shell exports from dpkg_runbuild_prepend
> >    dpkg: Remove builddeps install task
> >    sbuild: Add ccache support
> >    dpkg-base: Switch devshell to use schroot
> >    dpkg-base: Switch apt_fetch and apt_unpack to use schroot
> >    doc: Add sbuild-related documentation
> >    sbuild: Use .dsc file instead of source directory
> >    sbuild: Fixed proxy support
> >    sbuild: Fix debsrc_download for packages dependencies
> >
> >   RECIPE-API-CHANGELOG.md                       |  13 ++
> >   doc/user_manual.md                            |  22 ++-
> >   meta/classes/deb-dl-dir.bbclass               |   6 +-
> >   meta/classes/dpkg-base.bbclass                |  90 ++++++----
> >   meta/classes/dpkg-gbp.bbclass                 |  26 +--
> >   meta/classes/dpkg.bbclass                     | 109 +++++++++---
> >   meta/classes/sbuild.bbclass                   | 159 ++++++++++++++++++
> >   meta/conf/bitbake.conf                        |   2 +
> >   .../sbuild-chroot/sbuild-chroot-host.bb       |  13 ++
> >   .../sbuild-chroot/sbuild-chroot-target.bb     |  10 ++
> >   .../sbuild-chroot/sbuild-chroot.inc           |  39 +++++
> >   11 files changed, 408 insertions(+), 81 deletions(-)
> >   create mode 100644 meta/classes/sbuild.bbclass
> >   create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-
> host.bb
> >   create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-
> target.bb
> >   create mode 100644
> > meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
> >
> This is also just a rebase on the current next.
> We are planning to merge it in the next few days.

I'm happy to see the sbuild integrated into ISAR.
We run some of our layers against this series for a while now.
No issues so far (apart from changes required for custom tasks, etc...).

Felix

> 
> Bitbake update will be the first priority after that.
> 
> --
> You received this message because you are subscribed to the Google Groups
> "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to isar-users+unsubscribe@googlegroups.com.
> To view this discussion on the web visit
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.g
> oogle.com%2Fd%2Fmsgid%2Fisar-users%2F7a97d256-669e-f8a7-699b-
> e6418a270225%2540ilbers.de&amp;data=05%7C01%7Cfelix.moessbauer%40sie
> mens.com%7C17cda068ab944c38019808da58d37a4a%7C38ae3bcd95794fd4ad
> dab42e1495d55a%7C1%7C0%7C637919963283287304%7CUnknown%7CTWFpb
> GZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6M
> n0%3D%7C3000%7C%7C%7C&amp;sdata=VwmNjDEboXR4ZlDoHh71WtOjAvk1T
> jWTNMLfWqWllco%3D&amp;reserved=0.

Re: [PATCH v12 00/21] Sbuild/Schroot migration

[Baurzhan Ismagulov]

On Tue, Jun 28, 2022 at 07:24:30AM +0000, Moessbauer, Felix wrote:
> I'm happy to see the sbuild integrated into ISAR.
> We run some of our layers against this series for a while now.
> No issues so far (apart from changes required for custom tasks, etc...).

Thanks Felix for the good news! If we encounter no surprises, we're planning to
start tomorrow.

With kind regards,
Baurzhan

Re: [PATCH v12 00/21] Sbuild/Schroot migration

[Anton Mikanovich]

28.06.2022 09:48, Anton Mikanovich wrote:
> This patchset integrates sbuild/schroot tools into Isar build system.
>
> Base schroot images are created with sbuild-chroot-target or, in case
> of cross-build, sbuild-chroot-host recipes. These images are based on
> isar-bootstrap ones, but include some build-related stuff preinstalled.
>
> To use these changes you need to have sbuild and schroot installed and
> configured. User who runs build should be added to sbuild group.
>
> Currently, sbuild-related stuff is supported in latest kas.
>
> If Gilab CI is used, the following configuration changes are required:
> - external dir for schroot overlay should be set in the config
> /etc/gitlab-runner/config.toml:
>
> volumes = ["/path/to/overlay:/var/lib/schroot/union/overlay"]
>
> The patchset was tested with 'meta-iot2050' and 'xenomai-images'
> downstreams in combination with kas image mentioned above.
>
> Current limitations:
> - parallel building of several packages requires more free space in
> comparison with buildchroot-based architecture. This happens due
> the sbuild architecture which uses some basic layer (common for all
> packages) and per-package separate layer (where builddeps are downloaded
> and installed).
>
> Changes sinse v11:
> - Rebase on current next.
>
> Changes sinse v10:
> - Rebase on current next.
>
> Changes sinse v9:
> - Add caching improvement patch;
> - Remove per-package deb artifacts dir after usage to reduce disk space;
> - Improve schroot cleanup on failures.
>
> Changes since v8:
> - Added patch with support of package versions with epochs
>
> Changes since v7:
> - fixed work with proxy by adding some environment filters in sbuild;
> - DEB_BUILD_PROFILES support simplified and improved
> - Fixed gbp-based recipes rebuilding (get back --git-ignore-new option
> previously removed)
> - fixed base-apt caching for packages and source packages temporary
> installed in sbuild isolated environment
> - fixed rare build error related to ccache setup (added locking)
>
> Changes since v6:
> - added patch that switches sbuild from using source directory to using
> .dsc file. The file is generated before sbuild call by `dpkg-source`.
>
> Changes since v5:
> - warning about shell exports in dpkg_runbuild_prepend. This should
> avoided be reworked to templates.
>
> Changes since v4:
> - consider shell exports done in dpkg_runbuild_prepend and pass them
> to sbuild environment;
> - fixed ccache work.
>
> Changes since v3:
> - dpkg_do_mounts() and dpkg_undo_mounts() are not removed for downstreams
> compatibility;
> - dpkg_build_export is used for adjusting sbuild environment;
> - DEB_BUILD_PROFILES support for sbuild is fixed in case of cross-build.
>
> Changes since v2:
> - patches reworked/squashed for easier reading and understanding;
> - fixed building foreigh architectures with kas-docker;
> - implemented support of ccache;
> - fixed devshell and devshell_nodeps.
>
> Changes since v1:
> - parallel builds with different BUILD_DIR are supported;
> - parallel multiconfig targets in one build are supported;
> - per-task schroot configuration in /etc/schroot/ is now used;
> - patchset now passes Jenkins CI (so patches changes RFC => PATCH).
>
> Anton Mikanovich (3):
>    dpkg: Build packages with sbuild
>    dpkg-base: Cleanup on schroot fail
>    sbuild: Add changelog entry
>
> Felix Moessbauer (4):
>    fix: support build of packages with epoch version
>    always create apt-cache dirs in deb_dl_dir_import
>    avoid absolute SCHROOT_* paths to improve caching
>    locate generated binary package in WORKDIR
>
> Uladzimir Bely (14):
>    dpkg-gbp: Use separate command to export tarball
>    dpkg-gbp: Use host tools for dsc preparation
>    sbuild: Add recipes for host and target rootfs to run sbuild
>    sbuild: Introduce a class for another build method
>    sbuild: Support of DEB_BUILD_PROFILES
>    sbuild: Support of shell exports from dpkg_runbuild_prepend
>    dpkg: Remove builddeps install task
>    sbuild: Add ccache support
>    dpkg-base: Switch devshell to use schroot
>    dpkg-base: Switch apt_fetch and apt_unpack to use schroot
>    doc: Add sbuild-related documentation
>    sbuild: Use .dsc file instead of source directory
>    sbuild: Fixed proxy support
>    sbuild: Fix debsrc_download for packages dependencies
>
>   RECIPE-API-CHANGELOG.md                       |  13 ++
>   doc/user_manual.md                            |  22 ++-
>   meta/classes/deb-dl-dir.bbclass               |   6 +-
>   meta/classes/dpkg-base.bbclass                |  90 ++++++----
>   meta/classes/dpkg-gbp.bbclass                 |  26 +--
>   meta/classes/dpkg.bbclass                     | 109 +++++++++---
>   meta/classes/sbuild.bbclass                   | 159 ++++++++++++++++++
>   meta/conf/bitbake.conf                        |   2 +
>   .../sbuild-chroot/sbuild-chroot-host.bb       |  13 ++
>   .../sbuild-chroot/sbuild-chroot-target.bb     |  10 ++
>   .../sbuild-chroot/sbuild-chroot.inc           |  39 +++++
>   11 files changed, 408 insertions(+), 81 deletions(-)
>   create mode 100644 meta/classes/sbuild.bbclass
>   create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
>   create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
>   create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
>
Applied to next.

Re: [PATCH v12 08/21] dpkg: Remove builddeps install task

[Henning Schild]

Am Tue, 28 Jun 2022 09:49:00 +0300
schrieb Anton Mikanovich <amikan@ilbers.de>:

> From: Uladzimir Bely <ubely@ilbers.de>
> 
> While builddeps are now handled interlnally by sbuild, we don't need
> a task to preinstall them in buildchroot.
> 
> But for the second local (base-apt) build we need a way to keep
> all dependencies in form of .deb files in DL_DIR. This is done
> by executing additional commands in sbuild that copy them to/from
> an externally mounted folder.
> 
> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> ---
>  meta/classes/dpkg.bbclass   | 47
> ++++++++++++++++--------------------- meta/classes/sbuild.bbclass |
> 1 - 2 files changed, 20 insertions(+), 28 deletions(-)
> 
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> index 05b46c9..3c7b670 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -5,33 +5,6 @@ inherit dpkg-base
>  
>  PACKAGE_ARCH ?= "${DISTRO_ARCH}"
>  
> -# Install build dependencies for package
> -do_install_builddeps() {
> -    dpkg_do_mounts
> -    E="${@ isar_export_proxies(d)}"
> -    export DEB_BUILD_OPTIONS="${@ isar_deb_build_options(d)}"
> -    export DEB_BUILD_PROFILES="${@ isar_deb_build_profiles(d)}"
> -    distro="${DISTRO}"
> -    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
> -        distro="${HOST_DISTRO}"
> -    fi
> -
> -    deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
> -    sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
> -        ${PP}/${PPS} ${PACKAGE_ARCH} --download-only
> -    deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
> -    sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
> -        ${PP}/${PPS} ${PACKAGE_ARCH}
> -    dpkg_undo_mounts
> -}
> -
> -addtask install_builddeps after do_prepare_build before do_dpkg_build
> -do_install_builddeps[depends] += "${BUILDCHROOT_DEP}
> isar-apt:do_cache_config" -# apt and reprepro may not run in
> parallel, acquire the Isar lock -do_install_builddeps[lockfiles] +=
> "${REPO_ISAR_DIR}/isar.lock" -
> -addtask devshell after do_install_builddeps
> -
>  DPKG_PREBUILD_ENV_FILE="${WORKDIR}/dpkg_prebuild.env"
>  
>  do_prepare_build_append() {
> @@ -65,6 +38,16 @@ dpkg_runbuild() {
>          bbwarn "Export of '${line}' detected, please migrate to
> templates" done
>  
> +    distro="${DISTRO}"
> +    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
> +        distro="${HOST_DISTRO}"
> +    fi
> +
> +    deb_dl_dir_import "${WORKDIR}/rootfs" "${distro}"
> +
> +    deb_dir="/var/cache/apt/archives"
> +    ext_deb_dir="${PP}/rootfs/${deb_dir}"
> +
>      profiles="${@ isar_deb_build_profiles(d)}"
>      if [ ! -z "$profiles" ]; then
>          profiles=$(echo --profiles="$profiles" | sed -e 's/ \+/,/g')
> @@ -72,9 +55,19 @@ dpkg_runbuild() {
>  
>      export SBUILD_CONFIG="${SBUILD_CONFIG}"
>  
> +    echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
> +

This is not idempotent. If the task runs multiple times one gets many
of these lines in sbuild.conf.

Not sure why we keep that old file in the first place and do not simply
start over with a fresh file in every build.

Henning

>      sbuild -A -n -c ${SBUILD_CHROOT}
> --extra-repository="${ISAR_APT_REPO}" \ --host=${PACKAGE_ARCH}
> --build=${SBUILD_HOST_ARCH} ${profiles} \ --no-run-lintian
> --no-run-piuparts --no-run-autopkgtest \
> +        --chroot-setup-commands="cp -n --no-preserve=owner
> ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
> +        --finished-build-commands="rm -f
> ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
> +        --finished-build-commands="cp -n --no-preserve=owner
> ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
> --debbuildopts="--source-option=-I" \ --build-dir=${WORKDIR}
> ${WORKDIR}/${PPS} +
> +    deb_dl_dir_export "${WORKDIR}/rootfs" "${distro}"
> +
> +    # Cleanup apt artifacts
> +    sudo rm -rf ${WORKDIR}/rootfs
>  }
> diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
> index a3783bc..178ba1b 100644
> --- a/meta/classes/sbuild.bbclass
> +++ b/meta/classes/sbuild.bbclass
> @@ -8,7 +8,6 @@ SCHROOT_MOUNTS ?= ""
>  python __anonymous() {
>      import pwd
>      d.setVar('SCHROOT_USER', pwd.getpwuid(os.geteuid()).pw_name)
> -    d.setVar('SCHROOT_USER_HOME', pwd.getpwuid(os.geteuid()).pw_dir)
>  
>      mode = d.getVar('ISAR_CROSS_COMPILE', True)
>      distro_arch = d.getVar('DISTRO_ARCH')

Re: [PATCH v12 05/21] dpkg: Build packages with sbuild

[Jan Kiszka]

On 28.06.22 08:48, Anton Mikanovich wrote:
> Use previously exported to schroot rootfs to build the package with
> sbuild tool. Provide isar-apt as extra repository to be automatically
> add to apt sources.
> 
> Also added /home/.git-downloads mount while it's used
> as git alternates location.
> 
> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> ---
>  meta/classes/dpkg-base.bbclass | 31 ++++++++++++++++---------------
>  meta/classes/dpkg-gbp.bbclass  |  2 ++
>  meta/classes/dpkg.bbclass      | 10 ++++++++--
>  meta/classes/sbuild.bbclass    |  4 ++--
>  4 files changed, 28 insertions(+), 19 deletions(-)
> 
> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
> index 7e032ba..dbe35f4 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -4,6 +4,7 @@
>  #
>  # SPDX-License-Identifier: MIT
>  
> +inherit sbuild
>  inherit buildchroot
>  inherit debianize
>  inherit terminal
> @@ -16,6 +17,8 @@ DEPENDS_append_riscv64 = "${@' crossbuild-essential-riscv64' if d.getVar('ISAR_C
>  DEB_BUILD_PROFILES ?= ""
>  DEB_BUILD_OPTIONS ?= ""
>  
> +ISAR_APT_REPO ?= "deb [trusted=yes] file:///home/builder/${PN}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO} ${DEBDISTRONAME} main"
> +
>  python do_adjust_git() {
>      import subprocess
>  
> @@ -175,6 +178,7 @@ addtask prepare_build after do_patch do_transform_template before do_dpkg_build
>  # If Isar recipes depend on each other, they typically need the package
>  # deployed to isar-apt
>  do_prepare_build[deptask] = "do_deploy_deb"
> +do_prepare_build[depends] = "${SCHROOT_DEP}"
>  
>  BUILDROOT = "${BUILDCHROOT_DIR}/${PP}"
>  
> @@ -200,6 +204,15 @@ dpkg_undo_mounts() {
>      sudo rmdir ${BUILDROOT}
>  }
>  
> +do_prepare_build_append() {
> +    # Make a local copy of isar-apt repo that is not affected by other parallel builds
> +    mkdir -p ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
> +    rm -rf ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/*
> +    cp -Rl ${REPO_ISAR_DIR} ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
> +}
> +
> +do_prepare_build[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
> +
>  # Placeholder for actual dpkg_runbuild() implementation
>  dpkg_runbuild() {
>      die "This should never be called, overwrite it in your derived class"
> @@ -222,14 +235,11 @@ def isar_export_build_settings(d):
>      os.environ['DEB_BUILD_PROFILES'] = isar_deb_build_profiles(d)
>  
>  python do_dpkg_build() {
> -    lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
> -                             shared=True)
> -    bb.build.exec_func("dpkg_do_mounts", d)
> +    bb.build.exec_func('schroot_create_configs', d)
>      try:
>          bb.build.exec_func("dpkg_runbuild", d)
>      finally:
> -        bb.build.exec_func("dpkg_undo_mounts", d)
> -        bb.utils.unlockfile(lock)
> +        bb.build.exec_func('schroot_delete_configs', d)
>  }
>  
>  addtask dpkg_build
> @@ -259,7 +269,7 @@ python do_dpkg_build_setscene() {
>  addtask dpkg_build_setscene
>  do_dpkg_build_setscene[dirs] += "${S}/.."
>  
> -KEEP_INSTALLED_ON_CLEAN ?= "0"
> +do_dpkg_build[depends] = "${SCHROOT_DEP}"
>  
>  CLEANFUNCS += "deb_clean"
>  
> @@ -269,15 +279,6 @@ deb_clean() {
>          for d in ${DEBS}; do
>              repo_del_package "${REPO_ISAR_DIR}"/"${DISTRO}" \
>                  "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${d}"
> -            if [ "${KEEP_INSTALLED_ON_CLEAN}" = "1" ]; then
> -                continue;
> -            fi
> -            package=$(basename "${d}")
> -            package_remove="/usr/bin/apt-get remove -y ${package%%_*}"
> -            sudo -E chroot ${BUILDCHROOT_DIR} ${package_remove} || true
> -            if [ "${BUILDCHROOT_DIR}" != "${BUILDCHROOT_TARGET_DIR}" ]; then
> -                    sudo -E chroot ${BUILDCHROOT_TARGET_DIR} ${package_remove} || true
> -            fi
>          done
>      fi
>  }
> diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
> index 2b0b818..68172b3 100644
> --- a/meta/classes/dpkg-gbp.bbclass
> +++ b/meta/classes/dpkg-gbp.bbclass
> @@ -11,6 +11,8 @@ PATCHTOOL ?= "git"
>  
>  GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
>  
> +SCHROOT_MOUNTS = "${WORKDIR}:${PP} ${GITDIR}:/home/.git-downloads"
> +

This was always (at least in-tree) dead code because insert_mounts() was
never invoked. What was this setting here supposed to achieve?

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: [PATCH v12 05/21] dpkg: Build packages with sbuild

[Uladzimir Bely]

In the email from Wednesday, 11 January 2023 23:59:05 +03 user Jan Kiszka 
wrote:
> On 28.06.22 08:48, Anton Mikanovich wrote:
> > Use previously exported to schroot rootfs to build the package with
> > sbuild tool. Provide isar-apt as extra repository to be automatically
> > add to apt sources.
> > 
> > Also added /home/.git-downloads mount while it's used
> > as git alternates location.
> > 
> > Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> > Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> > ---
> > 
> >  meta/classes/dpkg-base.bbclass | 31 ++++++++++++++++---------------
> >  meta/classes/dpkg-gbp.bbclass  |  2 ++
> >  meta/classes/dpkg.bbclass      | 10 ++++++++--
> >  meta/classes/sbuild.bbclass    |  4 ++--
> >  4 files changed, 28 insertions(+), 19 deletions(-)
> > 
> > diff --git a/meta/classes/dpkg-base.bbclass
> > b/meta/classes/dpkg-base.bbclass index 7e032ba..dbe35f4 100644
> > --- a/meta/classes/dpkg-base.bbclass
> > +++ b/meta/classes/dpkg-base.bbclass
> > @@ -4,6 +4,7 @@
> > 
> >  #
> >  # SPDX-License-Identifier: MIT
> > 
> > +inherit sbuild
> > 
> >  inherit buildchroot
> >  inherit debianize
> >  inherit terminal
> > 
> > @@ -16,6 +17,8 @@ DEPENDS_append_riscv64 = "${@'
> > crossbuild-essential-riscv64' if d.getVar('ISAR_C> 
> >  DEB_BUILD_PROFILES ?= ""
> >  DEB_BUILD_OPTIONS ?= ""
> > 
> > +ISAR_APT_REPO ?= "deb [trusted=yes]
> > file:///home/builder/${PN}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO
> > } ${DEBDISTRONAME} main" +
> > 
> >  python do_adjust_git() {
> >  
> >      import subprocess
> > 
> > @@ -175,6 +178,7 @@ addtask prepare_build after do_patch
> > do_transform_template before do_dpkg_build> 
> >  # If Isar recipes depend on each other, they typically need the package
> >  # deployed to isar-apt
> >  do_prepare_build[deptask] = "do_deploy_deb"
> > 
> > +do_prepare_build[depends] = "${SCHROOT_DEP}"
> > 
> >  BUILDROOT = "${BUILDCHROOT_DIR}/${PP}"
> > 
> > @@ -200,6 +204,15 @@ dpkg_undo_mounts() {
> > 
> >      sudo rmdir ${BUILDROOT}
> >  
> >  }
> > 
> > +do_prepare_build_append() {
> > +    # Make a local copy of isar-apt repo that is not affected by other
> > parallel builds +    mkdir -p
> > ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
> > +    rm -rf ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/*
> > +    cp -Rl ${REPO_ISAR_DIR} ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
> > +}
> > +
> > +do_prepare_build[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
> > +
> > 
> >  # Placeholder for actual dpkg_runbuild() implementation
> >  dpkg_runbuild() {
> >  
> >      die "This should never be called, overwrite it in your derived class"
> > 
> > @@ -222,14 +235,11 @@ def isar_export_build_settings(d):
> >      os.environ['DEB_BUILD_PROFILES'] = isar_deb_build_profiles(d)
> >  
> >  python do_dpkg_build() {
> > 
> > -    lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
> > -                             shared=True)
> > -    bb.build.exec_func("dpkg_do_mounts", d)
> > +    bb.build.exec_func('schroot_create_configs', d)
> > 
> >      try:
> >          bb.build.exec_func("dpkg_runbuild", d)
> >      
> >      finally:
> > -        bb.build.exec_func("dpkg_undo_mounts", d)
> > -        bb.utils.unlockfile(lock)
> > +        bb.build.exec_func('schroot_delete_configs', d)
> > 
> >  }
> >  
> >  addtask dpkg_build
> > 
> > @@ -259,7 +269,7 @@ python do_dpkg_build_setscene() {
> > 
> >  addtask dpkg_build_setscene
> >  do_dpkg_build_setscene[dirs] += "${S}/.."
> > 
> > -KEEP_INSTALLED_ON_CLEAN ?= "0"
> > +do_dpkg_build[depends] = "${SCHROOT_DEP}"
> > 
> >  CLEANFUNCS += "deb_clean"
> > 
> > @@ -269,15 +279,6 @@ deb_clean() {
> > 
> >          for d in ${DEBS}; do
> >          
> >              repo_del_package "${REPO_ISAR_DIR}"/"${DISTRO}" \
> >              
> >                  "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}"
> >                  "${d}"
> > 
> > -            if [ "${KEEP_INSTALLED_ON_CLEAN}" = "1" ]; then
> > -                continue;
> > -            fi
> > -            package=$(basename "${d}")
> > -            package_remove="/usr/bin/apt-get remove -y ${package%%_*}"
> > -            sudo -E chroot ${BUILDCHROOT_DIR} ${package_remove} || true
> > -            if [ "${BUILDCHROOT_DIR}" != "${BUILDCHROOT_TARGET_DIR}" ];
> > then -                    sudo -E chroot ${BUILDCHROOT_TARGET_DIR}
> > ${package_remove} || true -            fi
> > 
> >          done
> >      
> >      fi
> >  
> >  }
> > 
> > diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
> > index 2b0b818..68172b3 100644
> > --- a/meta/classes/dpkg-gbp.bbclass
> > +++ b/meta/classes/dpkg-gbp.bbclass
> > @@ -11,6 +11,8 @@ PATCHTOOL ?= "git"
> > 
> >  GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
> > 
> > +SCHROOT_MOUNTS = "${WORKDIR}:${PP} ${GITDIR}:/home/.git-downloads"
> > +
> 
> This was always (at least in-tree) dead code because insert_mounts() was
> never invoked. What was this setting here supposed to achieve?
> 
> Jan

Hello.
You could look at these patches from early stage of isar/sbuild story

https://groups.google.com/g/isar-users/c/vNcgAkbdECI/m/NhROnwKOAgAJ
https://groups.google.com/g/isar-users/c/QBhw8FUwCe0/m/aDd3hY4oAwAJ

We didn't have proper "git alternates" solution at that time so mounting 
'.git' was used.

For now, these mounts might not be needed and we probably could remove these 
mountpoints and related code.