[PATCH v6 00/12] Sbuild/Schroot migration

[PATCH v6 00/12] Sbuild/Schroot migration

[Uladzimir Bely]

This is a patchset showing how sbuild/schroot tools can be integrated
into Isar build system.

This patchset should be applied on top of "preparation" patchset
named "Avoid using shell environment during the build".

Base schroot image is created with sbuild-chroot-target (or
sbuild-chroot-host in case of cross-build) recipe. These images are based
on isar-bootstrap ones, but include some build-related stuff preinstalled.

To use this changes you need to have sbuild and schroot installed and
configured. Current user should be added to sbuild group.

If 'kas-container' is used, it should be modified to support sbuild:
- /var/lib/schroot/union/overlay should be externally mounted (-v option)

Currently, sbuild-related stuff is supported in kas-isar:next image

To use the patchset with KAS:
- export KAS_IMAGE_VERSION=next

If 'gitlab' is used, the similar changes are required:
- `next` tag for kas image should be set
- external dir for schroot overlay should also be specified in
/etc/gitlab-runner/config.toml:
volumes = ["/path/to/overlay:/var/lib/schroot/union/overlay"]

The patchset was tested with 'meta-iot2050' and 'xenomai-images'
downstreams in combination with kas image mentioned above.

Current limitations:
- parallel building of several packages requires more free space in
comparison with buildchroot-based architecture. This happens due
the sbuild architecture which uses some basic layer (common for all
packages) and per-package separate layer (where builddeps are downloaded
and installed).

Changes since v5:
- warning about shell exports in dpkg_runbuild_prepend. This should
avoided be reworked to templates.

Changes since v4:
- consider shell exports done in dpkg_runbuild_prepend and pass them
to sbuild environment;
- fixed ccache work.

Changes since v3:
- dpkg_do_mounts() and dpkg_undo_mounts() are not removed for downstreams
compatibility;
- dpkg_build_export is used for adjusting sbuild environment;
- DEB_BUILD_PROFILES support for sbuild is fixed in case of cross-build.

Changes since v2:
- patches reworked/squashed for easier reading and understanding;
- fixed building foreigh architectures with kas-docker;
- implemented support of ccache;
- fixed devshell and devshell_nodeps,

Changes since v1:
- parallel builds with different BUILD_DIR are supported;
- parallel multiconfig targets in one build are supported;
- per-task schroot configuration in /etc/schroot/ is now used;
- patchset now passes Jenkins CI (so patches changes RFC => PATCH).



Anton Mikanovich (1):
  dpkg: Build packages with sbuild

Uladzimir Bely (11):
  dpkg-gbp: Use separate command to export tarball
  dpkg-gbp: Use host tools for dsc preparation
  sbuild: Add recipes for host and target rootfs to run sbuild
  sbuild: Introduce a class for another build method
  sbuild: support of DEB_BUILD_PROFILES
  sbuild: support of shell exports from dpkg_runbuild_prepend
  dpkg: Remove builddeps install task.
  sbuild: add ccache support
  dpkg-base: Switch devshell to use schroot
  dpkg-base: Switch apt_fetch and apt_unpack to use schroot
  doc: Add sbuild-related documentation

 doc/user_manual.md                            |  22 ++-
 meta/classes/dpkg-base.bbclass                |  79 ++++----
 meta/classes/dpkg-gbp.bbclass                 |  26 +--
 meta/classes/dpkg.bbclass                     |  88 ++++++---
 meta/classes/sbuild.bbclass                   | 184 ++++++++++++++++++
 meta/conf/bitbake.conf                        |   2 +
 .../sbuild-chroot/sbuild-chroot-host.bb       |  13 ++
 .../sbuild-chroot/sbuild-chroot-target.bb     |  10 +
 .../sbuild-chroot/sbuild-chroot.inc           |  39 ++++
 9 files changed, 382 insertions(+), 81 deletions(-)
 create mode 100644 meta/classes/sbuild.bbclass
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc

-- 
2.20.1

[PATCH v6 01/12] dpkg-gbp: Use separate command to export tarball

[Uladzimir Bely]

We don't actually build the package with gbp, but only prepare it for
the building with external dpkg-buildpackage command. In case there is
no need in real build we can perform only export which will produce
upstream tarball to pass for later building.
This allows to use any build tool for later processing.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-gbp.bbclass | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index d956e8c3..ddf23ca5 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -30,5 +30,9 @@ do_install_builddeps_append() {
 }
 
 dpkg_runbuild_prepend() {
-    export GBP_PREFIX="gbp buildpackage --git-ignore-new ${GBP_EXTRA_OPTIONS} --git-builder="
+    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+        sh -c "cd ${PP}/${PPS} && gbp buildpackage --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}"
+    # NOTE: `buildpackage --git-builder=/bin/true --git-pristine-tar` is used
+    # for compatibility with gbp version froms debian-stretch. In newer distros
+    # it's possible to use a subcommand `export-orig --pristine-tar`
 }
-- 
2.20.1

[PATCH v6 02/12] dpkg-gbp: Use host tools for dsc preparation

[Uladzimir Bely]

Instead of preinstalling gbp-related packages in sbuild chroot
we will use their host versions in order to keep sbuild environment
as clean as possible.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 doc/user_manual.md            |  4 +++-
 meta/classes/dpkg-gbp.bbclass | 24 ++++--------------------
 2 files changed, 7 insertions(+), 21 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 4d4fa763..df73ef89 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -80,7 +80,9 @@ apt install \
   qemu \
   qemu-user-static \
   reprepro \
-  sudo
+  sudo \
+  git-buildpackage \
+  pristine-tar
 ```
 
 If your host is >= buster, also install the following package.
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index ddf23ca5..2fc37b2f 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -9,29 +9,13 @@ S = "${WORKDIR}/git"
 
 PATCHTOOL ?= "git"
 
-GBP_DEPENDS ?= "git-buildpackage pristine-tar"
 GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
 
-do_install_builddeps_append() {
-    dpkg_do_mounts
-    distro="${DISTRO}"
-    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
-       distro="${HOST_DISTRO}"
-    fi
-    deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} \
-        apt-get install -y -o Debug::pkgProblemResolver=yes \
-                        --no-install-recommends --download-only ${GBP_DEPENDS}
-    deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} \
-        apt-get install -y -o Debug::pkgProblemResolver=yes \
-                        --no-install-recommends ${GBP_DEPENDS}
-    dpkg_undo_mounts
-}
-
 dpkg_runbuild_prepend() {
-    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
-        sh -c "cd ${PP}/${PPS} && gbp buildpackage --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}"
+    sh -c "
+        cd ${WORKDIR}/${PPS}
+        gbp buildpackage --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}
+    "
     # NOTE: `buildpackage --git-builder=/bin/true --git-pristine-tar` is used
     # for compatibility with gbp version froms debian-stretch. In newer distros
     # it's possible to use a subcommand `export-orig --pristine-tar`
-- 
2.20.1

[PATCH v6 03/12] sbuild: Add recipes for host and target rootfs to run sbuild

[Uladzimir Bely]

Similar to buildchroot, we need a separate rootfs to be used
for schroot.

It's based on bootstrapped rootfs, but includes several
common build-related packages.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 .../sbuild-chroot/sbuild-chroot-host.bb       | 13 +++++++
 .../sbuild-chroot/sbuild-chroot-target.bb     | 10 ++++++
 .../sbuild-chroot/sbuild-chroot.inc           | 36 +++++++++++++++++++
 3 files changed, 59 insertions(+)
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc

diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
new file mode 100644
index 00000000..aa82846b
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
@@ -0,0 +1,13 @@
+# Root filesystem for packages building
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2021 ilbers GmbH
+
+DESCRIPTION = "Isar sbuild/schroot filesystem for host"
+
+SBUILD_VARIANT = "host"
+
+require sbuild-chroot.inc
+
+ROOTFS_ARCH = "${HOST_ARCH}"
+ROOTFS_DISTRO = "${HOST_DISTRO}"
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
new file mode 100644
index 00000000..d75d783b
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
@@ -0,0 +1,10 @@
+# Root filesystem for packages building
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2021 ilbers GmbH
+
+DESCRIPTION = "Isar sbuild/schroot filesystem for target"
+
+SBUILD_VARIANT = "target"
+
+require sbuild-chroot.inc
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
new file mode 100644
index 00000000..177a8a6d
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -0,0 +1,36 @@
+# Common part for build chroot filesystem.
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2021 ilbers GmbH
+
+LICENSE = "gpl-2.0"
+LIC_FILES_CHKSUM = "file://${LAYERDIR_core}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
+
+PV = "1.0"
+
+inherit rootfs
+
+SBUILD_CHROOT_PREINSTALL_COMMON = " \
+    fakeroot \
+    build-essential \
+    debhelper \
+"
+
+SBUILD_CHROOT_PREINSTALL ?= " \
+    ${SBUILD_CHROOT_PREINSTALL_COMMON} \
+"
+
+SBUILD_CHROOT_DIR = "${WORKDIR}/rootfs"
+ROOTFSDIR = "${SBUILD_CHROOT_DIR}"
+ROOTFS_PACKAGES = "${SBUILD_CHROOT_PREINSTALL}"
+
+# We don't need /etc/apt/sources.list.d/isar-apt.list' while it's handled by sbuild
+ROOTFS_CONFIGURE_COMMAND_remove = "rootfs_configure_isar_apt"
+
+DEPLOY_SCHROOT = "${@d.getVar('SCHROOT_' + d.getVar('SBUILD_VARIANT').upper() + '_DIR')}"
+
+do_sbuildchroot_deploy[dirs] = "${DEPLOY_DIR}/schroot-${SBUILD_VARIANT}"
+do_sbuildchroot_deploy() {
+    ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_SCHROOT}"
+}
+addtask sbuildchroot_deploy before do_build after do_rootfs
-- 
2.20.1

[PATCH v6 04/12] sbuild: Introduce a class for another build method

[Uladzimir Bely]

This also adds mounts for base-apt inside schroot and adds import/export
deb files to/from schroot. So that it becomes possible to run second
 `cached` build from local base-apt repo.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/sbuild.bbclass | 160 ++++++++++++++++++++++++++++++++++++
 meta/conf/bitbake.conf      |   2 +
 2 files changed, 162 insertions(+)
 create mode 100644 meta/classes/sbuild.bbclass

diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
new file mode 100644
index 00000000..e4341a6f
--- /dev/null
+++ b/meta/classes/sbuild.bbclass
@@ -0,0 +1,160 @@
+# This software is a part of ISAR.
+# Copyright (C) 2021 ilbers GmbH
+
+SCHROOT_CONF ?= "/etc/schroot"
+
+SCHROOT_MOUNTS ?= ""
+
+python __anonymous() {
+    import pwd
+    d.setVar('SCHROOT_USER', pwd.getpwuid(os.geteuid()).pw_name)
+    d.setVar('SCHROOT_USER_HOME', pwd.getpwuid(os.geteuid()).pw_dir)
+
+    mode = d.getVar('ISAR_CROSS_COMPILE', True)
+    distro_arch = d.getVar('DISTRO_ARCH')
+    if mode == "0" or d.getVar('HOST_ARCH') ==  distro_arch or \
+       (d.getVar('HOST_DISTRO') == "debian-stretch" and distro_arch == "i386"):
+        d.setVar('SBUILD_HOST_ARCH', distro_arch)
+        d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_TARGET_DIR'))
+        dep = "sbuild-chroot-target:do_build"
+    else:
+        d.setVar('SBUILD_HOST_ARCH', d.getVar('HOST_ARCH'))
+        d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_HOST_DIR'))
+        dep = "sbuild-chroot-host:do_build"
+    d.setVar('SCHROOT_DEP', dep)
+}
+
+SBUILD_CHROOT ?= "${DEBDISTRONAME}-${SCHROOT_USER}-${@os.getpid()}"
+SBUILD_CHROOT_RW ?= "${SBUILD_CHROOT}-rw"
+
+SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}"
+SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}"
+SCHROOT_CONF_FILE_RW ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}-rw"
+
+schroot_create_configs() {
+    sudo -s <<'EOSUDO'
+        set -e
+
+        cat << EOF > "${SCHROOT_CONF_FILE}"
+[${SBUILD_CHROOT}]
+type=directory
+directory=${SCHROOT_DIR}
+profile=${SBUILD_CHROOT}
+users=${SCHROOT_USER}
+groups=root,sbuild
+root-users=${SCHROOT_USER}
+root-groups=root,sbuild
+source-root-users=${SCHROOT_USER}
+source-root-groups=root,sbuild
+union-type=overlay
+preserve-environment=true
+EOF
+
+        cat << EOF > "${SCHROOT_CONF_FILE_RW}"
+[${SBUILD_CHROOT_RW}]
+type=directory
+directory=${SCHROOT_DIR}
+profile=${SBUILD_CHROOT}
+users=${SCHROOT_USER}
+groups=root,sbuild
+root-users=${SCHROOT_USER}
+root-groups=root,sbuild
+preserve-environment=true
+EOF
+
+        mkdir -p "${SCHROOT_DIR}/etc/apt/preferences.d"
+        cat << EOF > "${SCHROOT_DIR}/etc/apt/preferences.d/isar-apt"
+Package: *
+Pin: release n=${DEBDISTRONAME}
+Pin-Priority: 1000
+EOF
+
+        # Prepare mount points
+        cp -rf "${SCHROOT_CONF}/sbuild" "${SBUILD_CONF_DIR}"
+        sbuild_fstab="${SBUILD_CONF_DIR}/fstab"
+
+        fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind 0 0"
+        grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}
+
+        if [ -d ${DL_DIR} ]; then
+            fstab_downloads="${DL_DIR} /downloads none rw,bind 0 0"
+            grep -qxF "${fstab_downloads}" ${sbuild_fstab} || echo "${fstab_downloads}" >> ${sbuild_fstab}
+        fi
+EOSUDO
+}
+
+schroot_delete_configs() {
+    sudo -s <<'EOSUDO'
+        set -e
+        if [ -d "${SBUILD_CONF_DIR}" ]; then
+            rm -rf "${SBUILD_CONF_DIR}"
+        fi
+        rm -f "${SCHROOT_DIR}/etc/apt/preferences.d/isar-apt"
+        rm -f "${SCHROOT_CONF_FILE}"
+        rm -f "${SCHROOT_CONF_FILE_RW}"
+EOSUDO
+}
+
+sbuild_export() {
+    SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
+    VAR=${1}; shift
+    VAR_LINE="'${VAR}' => '${@}',"
+    if [ -s "${SBUILD_CONFIG}" ]; then
+        sed -i -e "\$i\\" -e "${VAR_LINE}" ${SBUILD_CONFIG}
+    else
+        echo "\$build_environment = {" > ${SBUILD_CONFIG}
+        echo "${VAR_LINE}" >> ${SBUILD_CONFIG}
+        echo "};" >> ${SBUILD_CONFIG}
+    fi
+    export SBUILD_CONFIG="${SBUILD_CONFIG}"
+}
+
+schroot_install() {
+    schroot_create_configs
+    APTS="$1"
+
+    distro="${DISTRO}"
+    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+       distro="${HOST_DISTRO}"
+    fi
+
+    deb_dl_dir_import ${SCHROOT_DIR} ${distro}
+    schroot -d / -c ${SBUILD_CHROOT_RW} -u root -- \
+        apt install -y -o Debug::pkgProblemResolver=yes \
+                    --no-install-recommends --download-only ${APTS}
+    deb_dl_dir_export ${SCHROOT_DIR} ${distro}
+
+    schroot -d / -c ${SBUILD_CHROOT_RW} -u root -- \
+        apt install -y -o Debug::pkgProblemResolver=yes \
+                    --no-install-recommends ${APTS}
+    schroot_delete_configs
+}
+
+insert_mounts() {
+    sudo -s <<'EOSUDO'
+        set -e
+        for mp in ${SCHROOT_MOUNTS}; do
+            FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
+            grep -qxF "${FSTAB_LINE}" ${SBUILD_CONF_DIR}/fstab || \
+                echo "${FSTAB_LINE}" >> ${SBUILD_CONF_DIR}/fstab
+        done
+EOSUDO
+}
+
+remove_mounts() {
+    sudo -s <<'EOSUDO'
+        set -e
+        for mp in ${SCHROOT_MOUNTS}; do
+            FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
+            sed -i "\|${FSTAB_LINE}|d" ${SBUILD_CONF_DIR}/fstab
+        done
+EOSUDO
+}
+
+schroot_run() {
+    schroot_create_configs
+    insert_mounts
+    schroot $@
+    remove_mounts
+    schroot_delete_configs
+}
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index cd1c4a64..c1458f05 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -58,6 +58,8 @@ SSTATE_DIR ?= "${TOPDIR}/sstate-cache"
 SSTATE_MANIFESTS = "${TMPDIR}/sstate-control/${DISTRO}-${DISTRO_ARCH}"
 BUILDCHROOT_HOST_DIR = "${DEPLOY_DIR_BUILDCHROOT}-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
 BUILDCHROOT_TARGET_DIR = "${DEPLOY_DIR_BUILDCHROOT}-target/${DISTRO}-${DISTRO_ARCH}"
+SCHROOT_HOST_DIR = "${DEPLOY_DIR}/schroot-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
+SCHROOT_TARGET_DIR = "${DEPLOY_DIR}/schroot-target/${DISTRO}-${DISTRO_ARCH}"
 SDKCHROOT_DIR = "${DEPLOY_DIR_SDKCHROOT}/${DISTRO}-${DISTRO_ARCH}"
 CACHE = "${TMPDIR}/cache"
 KERNEL_FILE ?= "vmlinuz"
-- 
2.20.1

[PATCH v6 05/12] dpkg: Build packages with sbuild

[Uladzimir Bely]

From: Anton Mikanovich <amikan@ilbers.de>

Use previously exported to schroot rootfs to build the package with
sbuild tool. Provide isar-apt as extra repository to be automatically
add to apt sources.

Also added /home/.git-downloads mount while it's used
as git alternates location.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 31 ++++++++++++++++---------------
 meta/classes/dpkg-gbp.bbclass  |  2 ++
 meta/classes/dpkg.bbclass      | 10 ++++++++--
 meta/classes/sbuild.bbclass    |  4 ++--
 4 files changed, 28 insertions(+), 19 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 550bedfc..31d41754 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -4,6 +4,7 @@
 #
 # SPDX-License-Identifier: MIT
 
+inherit sbuild
 inherit buildchroot
 inherit debianize
 inherit terminal
@@ -16,6 +17,8 @@ DEPENDS_append_riscv64 = "${@' crossbuild-essential-riscv64' if d.getVar('ISAR_C
 DEB_BUILD_PROFILES ?= ""
 DEB_BUILD_OPTIONS ?= ""
 
+ISAR_APT_REPO ?= "deb [trusted=yes] file:///home/builder/${PN}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO} ${DEBDISTRONAME} main"
+
 python do_adjust_git() {
     import subprocess
 
@@ -173,6 +176,7 @@ addtask prepare_build after do_patch do_transform_template before do_dpkg_build
 # If Isar recipes depend on each other, they typically need the package
 # deployed to isar-apt
 do_prepare_build[deptask] = "do_deploy_deb"
+do_prepare_build[depends] = "${SCHROOT_DEP}"
 
 BUILDROOT = "${BUILDCHROOT_DIR}/${PP}"
 
@@ -198,6 +202,15 @@ dpkg_undo_mounts() {
     sudo rmdir ${BUILDROOT}
 }
 
+do_prepare_build_append() {
+    # Make a local copy of isar-apt repo that is not affected by other parallel builds
+    mkdir -p ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
+    rm -rf ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/*
+    cp -Rl ${REPO_ISAR_DIR} ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
+}
+
+do_prepare_build[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
+
 # Placeholder for actual dpkg_runbuild() implementation
 dpkg_runbuild() {
     die "This should never be called, overwrite it in your derived class"
@@ -220,14 +233,11 @@ def isar_export_build_settings(d):
     os.environ['DEB_BUILD_PROFILES'] = isar_deb_build_profiles(d)
 
 python do_dpkg_build() {
-    lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
-                             shared=True)
-    bb.build.exec_func("dpkg_do_mounts", d)
+    bb.build.exec_func('schroot_create_configs', d)
     try:
         bb.build.exec_func("dpkg_runbuild", d)
     finally:
-        bb.build.exec_func("dpkg_undo_mounts", d)
-        bb.utils.unlockfile(lock)
+        bb.build.exec_func('schroot_delete_configs', d)
 }
 
 addtask dpkg_build
@@ -259,7 +269,7 @@ python do_dpkg_build_setscene() {
 addtask dpkg_build_setscene
 do_dpkg_build_setscene[dirs] += "${S}/.."
 
-KEEP_INSTALLED_ON_CLEAN ?= "0"
+do_dpkg_build[depends] = "${SCHROOT_DEP}"
 
 CLEANFUNCS += "deb_clean"
 
@@ -269,15 +279,6 @@ deb_clean() {
         for d in ${DEBS}; do
             repo_del_package "${REPO_ISAR_DIR}"/"${DISTRO}" \
                 "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${d}"
-            if [ "${KEEP_INSTALLED_ON_CLEAN}" = "1" ]; then
-                continue;
-            fi
-            package=$(basename "${d}")
-            package_remove="/usr/bin/apt-get remove -y ${package%%_*}"
-            sudo -E chroot ${BUILDCHROOT_DIR} ${package_remove} || true
-            if [ "${BUILDCHROOT_DIR}" != "${BUILDCHROOT_TARGET_DIR}" ]; then
-                    sudo -E chroot ${BUILDCHROOT_TARGET_DIR} ${package_remove} || true
-            fi
         done
     fi
 }
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index 2fc37b2f..7eda9b03 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -11,6 +11,8 @@ PATCHTOOL ?= "git"
 
 GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
 
+SCHROOT_MOUNTS = "${WORKDIR}:${PP} ${GITDIR}:/home/.git-downloads"
+
 dpkg_runbuild_prepend() {
     sh -c "
         cd ${WORKDIR}/${PPS}
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 320102ba..848ffad1 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -39,6 +39,12 @@ dpkg_runbuild() {
     export DEB_BUILD_OPTIONS="${@ isar_deb_build_options(d)}"
     export DEB_BUILD_PROFILES="${@ isar_deb_build_profiles(d)}"
     export PARALLEL_MAKE="${PARALLEL_MAKE}"
-    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
-         /isar/build.sh ${PP}/${PPS} ${PACKAGE_ARCH}
+
+    export SBUILD_CONFIG="${SBUILD_CONFIG}"
+
+    sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
+        --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \
+        --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
+        --debbuildopts="--source-option=-I" \
+        --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
 }
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index e4341a6f..6fff664f 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -31,6 +31,8 @@ SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}"
 SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}"
 SCHROOT_CONF_FILE_RW ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}-rw"
 
+SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
+
 schroot_create_configs() {
     sudo -s <<'EOSUDO'
         set -e
@@ -96,7 +98,6 @@ EOSUDO
 }
 
 sbuild_export() {
-    SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
     VAR=${1}; shift
     VAR_LINE="'${VAR}' => '${@}',"
     if [ -s "${SBUILD_CONFIG}" ]; then
@@ -106,7 +107,6 @@ sbuild_export() {
         echo "${VAR_LINE}" >> ${SBUILD_CONFIG}
         echo "};" >> ${SBUILD_CONFIG}
     fi
-    export SBUILD_CONFIG="${SBUILD_CONFIG}"
 }
 
 schroot_install() {
-- 
2.20.1

[PATCH v6 06/12] sbuild: support of DEB_BUILD_PROFILES

[Uladzimir Bely]

In Isar packages may use their own profiles. For example, linux-custom.inc
introduces 'nolibcdev' profile that should be passed to dpkg-buildpackage
in both cross and native builds.

By default, sbuild for cross-building uses passes "cross,nocheck" options
to dpkg-buildpackage and ignores DEB_BUILD_PROFILES environment variable.

This change makes sbuild use custom profiles even in cross-build mode.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 848ffad1..5511de64 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -40,10 +40,18 @@ dpkg_runbuild() {
     export DEB_BUILD_PROFILES="${@ isar_deb_build_profiles(d)}"
     export PARALLEL_MAKE="${PARALLEL_MAKE}"
 
+    profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 | cut -d "'" -f 4)
+    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+        profiles="${profiles} cross nocheck"
+    fi
+    if [ ! -z "$profiles" ]; then
+        profiles=$(echo --profiles="$profiles" | sed -e 's/ \+/,/g')
+    fi
+
     export SBUILD_CONFIG="${SBUILD_CONFIG}"
 
     sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
-        --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \
+        --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
         --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
         --debbuildopts="--source-option=-I" \
         --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
-- 
2.20.1

[PATCH v6 07/12] sbuild: support of shell exports from dpkg_runbuild_prepend

[Uladzimir Bely]

Many of recipes often use shell exports done in dpkg_run_prepend.

While sbuild is performed in isolated environment, we need a way to pass
these exported variables to it. This is done by storing environment just
before dpkg_runbuild (after prepare_build) and finding just before
the actual build what has been changed or added.

This patch allows using shell exports, but warning will be produced.
It's recommended to modify the code (in downstreams) and replace shell
exports by other methods, like using templates.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 5511de64..82a722df 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -32,6 +32,12 @@ do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
 
 addtask devshell after do_install_builddeps
 
+DPKG_PREBUILD_ENV_FILE="${WORKDIR}/dpkg_prebuild.env"
+
+do_prepare_build_append() {
+    env > ${DPKG_PREBUILD_ENV_FILE}
+}
+
 # Build package from sources using build script
 dpkg_runbuild() {
     E="${@ isar_export_proxies(d)}"
@@ -40,6 +46,27 @@ dpkg_runbuild() {
     export DEB_BUILD_PROFILES="${@ isar_deb_build_profiles(d)}"
     export PARALLEL_MAKE="${PARALLEL_MAKE}"
 
+    env | while read -r line; do
+        # Filter the same lines
+        grep -q "^${line}" ${DPKG_PREBUILD_ENV_FILE} && continue
+        # Filter some standard variables
+        echo ${line} | grep -q "^HOME=" && continue
+        echo ${line} | grep -q "^PWD=" && continue
+
+        var=$(echo "${line}" | cut -d '=' -f1)
+        value=$(echo "${line}" | cut -d '=' -f2-)
+        sbuild_export $var "$value"
+
+        # Don't warn some variables
+        [ "${var}" = "PARALLEL_MAKE" ] && continue
+        [ "${var}" = "CCACHE_DIR" ] && continue
+        [ "${var}" = "PATH_PREPEND" ] && continue
+        [ "${var}" = "DEB_BUILD_OPTIONS" ] && continue
+        [ "${var}" = "DEB_BUILD_PROFILES" ] && continue
+
+        bbwarn "Export of '${line}' detected, please migrate to templates"
+    done
+
     profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 | cut -d "'" -f 4)
     if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
         profiles="${profiles} cross nocheck"
-- 
2.20.1

[PATCH v6 08/12] dpkg: Remove builddeps install task.

[Uladzimir Bely]

While builddeps are now handled interlnally by sbuild, we don't need
a task to preinstall them in buildchroot.

But for the second local (base-apt) build we need a way to keep
all dependencies in form of .deb files in DL_DIR. This is done
by executing additional commands in sbuild that copy them to/from
an externally mounted folder.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass | 47 +++++++++++++++++----------------------
 1 file changed, 20 insertions(+), 27 deletions(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 82a722df..4cf405ad 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -5,33 +5,6 @@ inherit dpkg-base
 
 PACKAGE_ARCH ?= "${DISTRO_ARCH}"
 
-# Install build dependencies for package
-do_install_builddeps() {
-    dpkg_do_mounts
-    E="${@ isar_export_proxies(d)}"
-    export DEB_BUILD_OPTIONS="${@ isar_deb_build_options(d)}"
-    export DEB_BUILD_PROFILES="${@ isar_deb_build_profiles(d)}"
-    distro="${DISTRO}"
-    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
-        distro="${HOST_DISTRO}"
-    fi
-
-    deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
-        ${PP}/${PPS} ${PACKAGE_ARCH} --download-only
-    deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
-        ${PP}/${PPS} ${PACKAGE_ARCH}
-    dpkg_undo_mounts
-}
-
-addtask install_builddeps after do_prepare_build before do_dpkg_build
-do_install_builddeps[depends] += "isar-apt:do_cache_config"
-# apt and reprepro may not run in parallel, acquire the Isar lock
-do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
-
-addtask devshell after do_install_builddeps
-
 DPKG_PREBUILD_ENV_FILE="${WORKDIR}/dpkg_prebuild.env"
 
 do_prepare_build_append() {
@@ -67,6 +40,21 @@ dpkg_runbuild() {
         bbwarn "Export of '${line}' detected, please migrate to templates"
     done
 
+    distro="${DISTRO}"
+    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+        distro="${HOST_DISTRO}"
+    fi
+
+    deb_dl_dir_import "${WORKDIR}/rootfs" "${distro}"
+
+    deb_dir="/var/cache/apt/archives/"
+    ext_deb_dir="/home/builder/${PN}/rootfs/${deb_dir}"
+
+    ( flock 9
+        grep -qxF '$apt_keep_downloaded_packages = 1;' ${SCHROOT_USER_HOME}/.sbuildrc ||
+            echo '$apt_keep_downloaded_packages = 1;' >> ${SCHROOT_USER_HOME}/.sbuildrc
+    ) 9>"${TMPDIR}/sbuildrc.lock"
+
     profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 | cut -d "'" -f 4)
     if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
         profiles="${profiles} cross nocheck"
@@ -80,6 +68,11 @@ dpkg_runbuild() {
     sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
         --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
         --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
+        --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
+        --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
+        --finished-build-commands="cp -n --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
         --debbuildopts="--source-option=-I" \
         --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
+
+    deb_dl_dir_export "${WORKDIR}/rootfs" "${distro}"
 }
-- 
2.20.1

[PATCH v6 09/12] sbuild: add ccache support

[Uladzimir Bely]

This adds ccache support for custom packages in the same manner
as it was done previously for buildchroot-based build, by using
USE_CCACHE variable in local.conf or per-recipe.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass                     |  4 ++++
 meta/classes/sbuild.bbclass                   | 21 +++++++++++++++++++
 .../sbuild-chroot/sbuild-chroot.inc           |  1 +
 3 files changed, 26 insertions(+)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 4cf405ad..3cd40ba8 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -55,6 +55,10 @@ dpkg_runbuild() {
             echo '$apt_keep_downloaded_packages = 1;' >> ${SCHROOT_USER_HOME}/.sbuildrc
     ) 9>"${TMPDIR}/sbuildrc.lock"
 
+    if [ ${USE_CCACHE} -eq 1 ]; then
+        schroot_configure_ccache
+    fi
+
     profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 | cut -d "'" -f 4)
     if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
         profiles="${profiles} cross nocheck"
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 6fff664f..9896b772 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -158,3 +158,24 @@ schroot_run() {
     remove_mounts
     schroot_delete_configs
 }
+
+schroot_configure_ccache() {
+    sudo -s <<'EOSUDO'
+        set -e
+
+        sbuild_fstab="${SBUILD_CONF_DIR}/fstab"
+
+        install --group=sbuild --mode=2775 -d ${CCACHE_DIR}
+        fstab_ccachedir="${CCACHE_DIR} /ccache none rw,bind 0 0"
+        grep -qxF "${fstab_ccachedir}" ${sbuild_fstab} || echo "${fstab_ccachedir}" >> ${sbuild_fstab}
+
+        cat << END > ${CCACHE_DIR}/sbuild-setup
+#!/bin/sh
+export PATH="\$PATH_PREPEND:\$PATH"
+exec "\$@"
+END
+        chmod a+rx ${CCACHE_DIR}/sbuild-setup
+
+        echo "command-prefix=/ccache/sbuild-setup" >> "${SCHROOT_CONF_FILE}"
+EOSUDO
+}
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
index 177a8a6d..74811e22 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -14,6 +14,7 @@ SBUILD_CHROOT_PREINSTALL_COMMON = " \
     fakeroot \
     build-essential \
     debhelper \
+    ccache \
 "
 
 SBUILD_CHROOT_PREINSTALL ?= " \
-- 
2.20.1

[PATCH v6 10/12] dpkg-base: Switch devshell to use schroot

[Uladzimir Bely]

While packages are now build in schroot with sbuild, devshell
task should make terminal open inside sbuild environment.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-base.bbclass                | 23 ++++++++++++++-----
 meta/classes/sbuild.bbclass                   |  3 +++
 .../sbuild-chroot/sbuild-chroot.inc           |  2 ++
 3 files changed, 22 insertions(+), 6 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 31d41754..238a0bbd 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -296,19 +296,30 @@ do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
 do_deploy_deb[dirs] = "${S}"
 
 python do_devshell() {
-    bb.build.exec_func('dpkg_do_mounts', d)
+    bb.build.exec_func('schroot_create_configs', d)
 
     isar_export_proxies(d)
     isar_export_ccache(d)
     isar_export_build_settings(d)
 
-    buildchroot = d.getVar('BUILDCHROOT_DIR')
+    schroot = d.getVar('SBUILD_CHROOT')
+    isar_apt = d.getVar('ISAR_APT_REPO')
     pp_pps = os.path.join(d.getVar('PP'), d.getVar('PPS'))
-    # the PATH variable is not forwarded by sudo -E.
-    termcmd = "sudo -E chroot {0} sh -c 'cd {1}; export PATH=$PATH_PREPEND:$PATH; $SHELL -i'"
-    oe_terminal(termcmd.format(buildchroot, pp_pps), "Isar devshell", d)
 
-    bb.build.exec_func('dpkg_undo_mounts', d)
+    install_deps = ":" if d.getVar('BB_CURRENTTASK') == "devshell_nodeps" else "mk-build-deps -i -t \
+        \"apt-get -y -q -o Debug::pkgProblemResolver=yes --no-install-recommends --allow-downgrades\" \
+        debian/control"
+
+    termcmd = "schroot -d / -c {0} -u root -- sh -c ' \
+        cd {1}; \
+        echo {2} > /etc/apt/sources.list.d/isar_apt.list; \
+        apt-get -y -q update; \
+        {3}; \
+        $SHELL -i \
+    '"
+    oe_terminal(termcmd.format(schroot, pp_pps, isar_apt, install_deps), "Isar devshell", d)
+
+    bb.build.exec_func('schroot_delete_configs', d)
 }
 
 addtask devshell after do_prepare_build
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 9896b772..eb358c71 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -78,6 +78,9 @@ EOF
         fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind 0 0"
         grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}
 
+        fstab_pkgdir="${WORKDIR} /home/builder/${PN} none rw,bind 0 0"
+        grep -qxF "${fstab_pkgdir}" ${sbuild_fstab} || echo "${fstab_pkgdir}" >> ${sbuild_fstab}
+
         if [ -d ${DL_DIR} ]; then
             fstab_downloads="${DL_DIR} /downloads none rw,bind 0 0"
             grep -qxF "${fstab_downloads}" ${sbuild_fstab} || echo "${fstab_downloads}" >> ${sbuild_fstab}
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
index 74811e22..4688da1d 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -15,6 +15,8 @@ SBUILD_CHROOT_PREINSTALL_COMMON = " \
     build-essential \
     debhelper \
     ccache \
+    devscripts \
+    equivs \
 "
 
 SBUILD_CHROOT_PREINSTALL ?= " \
-- 
2.20.1

[PATCH v6 11/12] dpkg-base: Switch apt_fetch and apt_unpack to use schroot

[Uladzimir Bely]

This moves downloading and unpacking deb-src to schroot environment.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 25 +++++++++----------------
 1 file changed, 9 insertions(+), 16 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 238a0bbd..7220f64f 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -101,19 +101,12 @@ python() {
 }
 
 do_apt_fetch() {
-    dpkg_do_mounts
-    E="${@ isar_export_proxies(d)}"
-    sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update \
-        -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
-        -o Dir::Etc::SourceParts="-" \
-        -o APT::Get::List-Cleanup="0"
-
+    schroot_create_configs
     for uri in "${SRC_APT}"; do
-        sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+        schroot -d / -c ${SBUILD_CHROOT} -- \
             sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only-source source "$2"' my_script "${DISTRO}" "${uri}"
     done
-
-    dpkg_undo_mounts
+    schroot_delete_configs
 }
 
 addtask apt_fetch after do_unpack before do_apt_unpack
@@ -122,13 +115,14 @@ do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
 # Add dependency from the correct buildchroot: host or target
 do_apt_fetch[depends] = "${BUILDCHROOT_DEP}"
 
+# Add dependency from the correct schroot: host or target
+do_apt_fetch[depends] += "${SCHROOT_DEP}"
+
 do_apt_unpack() {
     rm -rf ${S}
-    dpkg_do_mounts
-    E="${@ isar_export_proxies(d)}"
-
+    schroot_create_configs
     for uri in "${SRC_APT}"; do
-        sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+        schroot -d / -c ${SBUILD_CHROOT} -- \
             sh -c ' \
                 set -e
                 dscfile="$(apt-get -y -qq --print-uris --only-source source "${2}" | cut -d " " -f2 | grep -E "*.dsc")"
@@ -137,8 +131,7 @@ do_apt_unpack() {
                 dpkg-source -x "${dscfile}" "${PPS}"' \
                     my_script "${DISTRO}" "${uri}"
     done
-
-    dpkg_undo_mounts
+    schroot_delete_configs
 }
 
 addtask apt_unpack after do_apt_fetch before do_patch
-- 
2.20.1

[PATCH v6 12/12] doc: Add sbuild-related documentation

[Uladzimir Bely]

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 doc/user_manual.md | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index df73ef89..966e7125 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -85,6 +85,24 @@ apt install \
   pristine-tar
 ```
 
+Additional setup is required since `sbuild` is now used for package build.
+Install the following packages:
+```
+apt install \
+  sbuild \
+  schroot
+```
+Also, user who runs isar should be added to `sbuild` group.
+
+**NOTE:** sbuild version (<=0.78.1) packaged in Debian Buster doesn't support
+`$apt_keep_downloaded_packages` option which is required in Isar for
+populating `${DL_DIR}/deb`. So, host `sbuild` in this case should be manually
+upgraded to >=0.81.2 version from Debian Bullseye.
+
+```
+sudo gpasswd -a <username> sbuild
+```
+
 If your host is >= buster, also install the following package.
 ```
 apt install python3-distutils
-- 
2.20.1

RE: [PATCH v6 07/12] sbuild: support of shell exports from dpkg_runbuild_prepend

[Moessbauer, Felix]

Hi,

> +
> +        # Don't warn some variables
> +        [ "${var}" = "PARALLEL_MAKE" ] && continue
> +        [ "${var}" = "CCACHE_DIR" ] && continue
> +        [ "${var}" = "PATH_PREPEND" ] && continue
> +        [ "${var}" = "DEB_BUILD_OPTIONS" ] && continue
> +        [ "${var}" = "DEB_BUILD_PROFILES" ] && continue
> +
> +        bbwarn "Export of '${line}' detected, please migrate to templates"

There are more variables that have to be excluded, e.g. all proxy related settings:
(http_proxy, HTTP_PROXY, https_proxy, HTTPS_PROXY, no_proxy, NO_PROXY).

But the proxy settings have to be passed into the sbuild environment.
Currently, building behind a proxy fails (in do_dpkg_build):

+------------------------------------------------------------------------------+
| Update chroot                                                                |
+------------------------------------------------------------------------------+
[...]
Err:8 http://deb.debian.org/debian bullseye Release
  Cannot initiate the connection to deb.debian.org:80 (2a04:4e42:62::644). - connect (101: Network is unreachable)

Best regards,
Felix

--
Siemens AG, Technology, T CED SES-DE
Otto-Hahn-Ring 6, 81739 München, Germany

RE: [PATCH v6 07/12] sbuild: support of shell exports from dpkg_runbuild_prepend

[Moessbauer, Felix]

[-- Attachment #1: Type: text/plain, Size: 3128 bytes --]

Hi,

The attached patch adds support to run the sbuild behind a proxy.
I briefly tested this with "bitbake mc:qemuamd64-bullseye:isar-image-base" in an environment where proxies are mandatory.

Best regards,
Felix

> -----Original Message-----
> From: isar-users@googlegroups.com <isar-users@googlegroups.com> On
> Behalf Of Moessbauer, Felix
> Sent: Thursday, March 3, 2022 2:37 PM
> To: Uladzimir Bely <ubely@ilbers.de>
> Cc: isar-users@googlegroups.com
> Subject: RE: [PATCH v6 07/12] sbuild: support of shell exports from
> dpkg_runbuild_prepend
> 
> Hi,
> 
> > +
> > +        # Don't warn some variables
> > +        [ "${var}" = "PARALLEL_MAKE" ] && continue
> > +        [ "${var}" = "CCACHE_DIR" ] && continue
> > +        [ "${var}" = "PATH_PREPEND" ] && continue
> > +        [ "${var}" = "DEB_BUILD_OPTIONS" ] && continue
> > +        [ "${var}" = "DEB_BUILD_PROFILES" ] && continue
> > +
> > +        bbwarn "Export of '${line}' detected, please migrate to templates"
> 
> There are more variables that have to be excluded, e.g. all proxy related
> settings:
> (http_proxy, HTTP_PROXY, https_proxy, HTTPS_PROXY, no_proxy, NO_PROXY).
> 
> But the proxy settings have to be passed into the sbuild environment.
> Currently, building behind a proxy fails (in do_dpkg_build):
> 
> +------------------------------------------------------------------------------+
> | Update chroot                                                                |
> +------------------------------------------------------------------------------+
> [...]
> Err:8
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdeb.debia
> n.org%2Fdebian&amp;data=04%7C01%7Cfelix.moessbauer%40siemens.com%7
> C2c6b2941e0eb40ffccb408d9fd1ad566%7C38ae3bcd95794fd4addab42e1495d5
> 5a%7C1%7C0%7C637819114851078826%7CUnknown%7CTWFpbGZsb3d8eyJWI
> joiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C300
> 0&amp;sdata=aLGsi%2BBgf28Cm5g7GGhBY5h3eRu7cL5do5Ern45lsVA%3D&am
> p;reserved=0 bullseye Release
>   Cannot initiate the connection to deb.debian.org:80 (2a04:4e42:62::644). -
> connect (101: Network is unreachable)
> 
> Best regards,
> Felix
> 
> --
> Siemens AG, Technology, T CED SES-DE
> Otto-Hahn-Ring 6, 81739 München, Germany
> 
> 
> --
> You received this message because you are subscribed to the Google Groups
> "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to isar-users+unsubscribe@googlegroups.com.
> To view this discussion on the web visit
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.g
> oogle.com%2Fd%2Fmsgid%2Fisar-
> users%2FAM9PR10MB48698F892FBA3A33D2C0405289049%2540AM9PR10MB4
> 869.EURPRD10.PROD.OUTLOOK.COM&amp;data=04%7C01%7Cfelix.moessbau
> er%40siemens.com%7C2c6b2941e0eb40ffccb408d9fd1ad566%7C38ae3bcd957
> 94fd4addab42e1495d55a%7C1%7C0%7C637819114851078826%7CUnknown%7
> CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJ
> XVCI6Mn0%3D%7C3000&amp;sdata=PGDjMxGm4JkrqZoktZXAJakrj3Xmxt1w8Gq
> vVVYmL5w%3D&amp;reserved=0.

[-- Attachment #2: 0001-export-proxies-for-apt-in-sbuild-schroot.patch --]
[-- Type: application/octet-stream, Size: 2390 bytes --]

From 669d14a8b6285f45782b3969a5cea1737d331e2e Mon Sep 17 00:00:00 2001
From: Felix Moessbauer <felix.moessbauer@siemens.com>
Date: Wed, 9 Mar 2022 18:38:24 +0100
Subject: [PATCH 1/1] export proxies for apt in sbuild / schroot

The proxy variables have to be manually set for the apt
inside the schroot.
Otherwise, the apt-update that is run as part of sbuild
fails to fetch dependencies.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/dpkg.bbclass   |  3 +++
 meta/classes/sbuild.bbclass | 18 ++++++++++++++++++
 2 files changed, 21 insertions(+)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 314ac86e..cb5d4139 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -34,6 +34,9 @@ dpkg_runbuild() {
         [ "${var}" = "CCACHE_DIR" ] && continue
         [ "${var}" = "PATH_PREPEND" ] && continue
         [ "${var}" = "DEB_BUILD_OPTIONS" ] && continue
+        [ "${var}" = "http_proxy" ] && continue
+        [ "${var}" = "https_proxy" ] && continue
+        [ "${var}" = "no_proxy" ] && continue
 
         bbwarn "Export of '${line}' detected, please migrate to templates"
     done
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index eb358c71..7d2b8ac7 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -33,6 +33,20 @@ SCHROOT_CONF_FILE_RW ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}-rw"
 
 SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
 
+def get_apt_proxy_config(d):
+    isar_export_proxies(d)
+    out = ''
+    if 'http_proxy' in os.environ:
+        out += 'Acquire::http::Proxy "{}";\n'.format(os.environ['http_proxy'])
+    if 'https_proxy' in os.environ:
+        out += 'Acquire::https::Proxy "{}";\n'.format(os.environ['https_proxy'])
+    if 'no_proxy' in os.environ:
+        no_proxies = os.environ['no_proxy'].split(',')
+        for np in no_proxies:
+            out += 'Acquire::http::Proxy::{} DIRECT;\n'.format(np.strip())
+            out += 'Acquire::https::Proxy::{} DIRECT;\n'.format(np.strip())
+    return out
+
 schroot_create_configs() {
     sudo -s <<'EOSUDO'
         set -e
@@ -69,6 +83,10 @@ EOF
 Package: *
 Pin: release n=${DEBDISTRONAME}
 Pin-Priority: 1000
+EOF
+
+        cat << EOF > "${SCHROOT_DIR}/etc/apt/apt.conf.d/proxy.conf"
+${@ get_apt_proxy_config(d) }
 EOF
 
         # Prepare mount points
-- 
2.30.2

Re: [PATCH v6 07/12] sbuild: support of shell exports from dpkg_runbuild_prepend

[Uladzimir Bely]

In the email from Wednesday, 9 March 2022 20:57:32 +03 user Moessbauer, Felix 
wrote:
> Hi,
> 
> The attached patch adds support to run the sbuild behind a proxy.
> I briefly tested this with "bitbake mc:qemuamd64-bullseye:isar-image-base"
> in an environment where proxies are mandatory.
> 

Hi.

I've fixed proxy in a bit different way. I found that sbuild supports so 
called 'environment filters' that allow to pass particualr variables to the 
build environment.

You can look at the patch https://github.com/ilbers/isar/commit/380e17d9 
which, besides fixing proxy, also modifies a bit `sbuild_export()` function, 
so that no additional same lines would added to sbuild.conf on the sequential 
runs of build.

I'm not completely sure, but it seems that if some variable is added to 
$environment_filter array, it even doesn't need to be passed via 
sbuild_export(). But this should be tested

> Best regards,
> Felix
> 
> > -----Original Message-----
> > From: isar-users@googlegroups.com <isar-users@googlegroups.com> On
> > Behalf Of Moessbauer, Felix
> > Sent: Thursday, March 3, 2022 2:37 PM
> > To: Uladzimir Bely <ubely@ilbers.de>
> > Cc: isar-users@googlegroups.com
> > Subject: RE: [PATCH v6 07/12] sbuild: support of shell exports from
> > dpkg_runbuild_prepend
> > 
> > Hi,
> > 
> > > +
> > > +        # Don't warn some variables
> > > +        [ "${var}" = "PARALLEL_MAKE" ] && continue
> > > +        [ "${var}" = "CCACHE_DIR" ] && continue
> > > +        [ "${var}" = "PATH_PREPEND" ] && continue
> > > +        [ "${var}" = "DEB_BUILD_OPTIONS" ] && continue
> > > +        [ "${var}" = "DEB_BUILD_PROFILES" ] && continue
> > > +
> > > +        bbwarn "Export of '${line}' detected, please migrate to
> > > templates"
> > 
> > There are more variables that have to be excluded, e.g. all proxy related
> > settings:
> > (http_proxy, HTTP_PROXY, https_proxy, HTTPS_PROXY, no_proxy, NO_PROXY).
> > 
> > But the proxy settings have to be passed into the sbuild environment.
> > Currently, building behind a proxy fails (in do_dpkg_build):
> > 
> > +-------------------------------------------------------------------------
> > -----+> 
> > | Update chroot                                                           
> > |     |> 
> > +-------------------------------------------------------------------------
> > -----+ [...]
> > Err:8
> > https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdeb.debia
> > n.org%2Fdebian&amp;data=04%7C01%7Cfelix.moessbauer%40siemens.com%7
> > C2c6b2941e0eb40ffccb408d9fd1ad566%7C38ae3bcd95794fd4addab42e1495d5
> > 5a%7C1%7C0%7C637819114851078826%7CUnknown%7CTWFpbGZsb3d8eyJWI
> > joiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C300
> > 0&amp;sdata=aLGsi%2BBgf28Cm5g7GGhBY5h3eRu7cL5do5Ern45lsVA%3D&am
> > p;reserved=0 bullseye Release
> > 
> >   Cannot initiate the connection to deb.debian.org:80 (2a04:4e42:62::644).
> >   -
> > 
> > connect (101: Network is unreachable)
> > 
> > Best regards,
> > Felix
> > 
> > --
> > Siemens AG, Technology, T CED SES-DE
> > Otto-Hahn-Ring 6, 81739 München, Germany
> > 
> > 
> > --
> > You received this message because you are subscribed to the Google Groups
> > "isar-users" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to isar-users+unsubscribe@googlegroups.com.
> > To view this discussion on the web visit
> > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.g
> > oogle.com%2Fd%2Fmsgid%2Fisar-
> > users%2FAM9PR10MB48698F892FBA3A33D2C0405289049%2540AM9PR10MB4
> > 869.EURPRD10.PROD.OUTLOOK.COM&amp;data=04%7C01%7Cfelix.moessbau
> > er%40siemens.com%7C2c6b2941e0eb40ffccb408d9fd1ad566%7C38ae3bcd957
> > 94fd4addab42e1495d55a%7C1%7C0%7C637819114851078826%7CUnknown%7
> > CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJ
> > XVCI6Mn0%3D%7C3000&amp;sdata=PGDjMxGm4JkrqZoktZXAJakrj3Xmxt1w8Gq
> > vVVYmL5w%3D&amp;reserved=0.


-- 
Uladzimir Bely

RE: [PATCH v6 07/12] sbuild: support of shell exports from dpkg_runbuild_prepend

[Moessbauer, Felix]

Hi,

> -----Original Message-----
> From: Uladzimir Bely <ubely@ilbers.de>
> Sent: Monday, March 14, 2022 1:29 PM
> To: isar-users@googlegroups.com
> Cc: isar-users@googlegroups.com; Kiszka, Jan (T CED)
> <jan.kiszka@siemens.com>; Moessbauer, Felix (T CED SES-DE)
> <felix.moessbauer@siemens.com>
> Subject: Re: [PATCH v6 07/12] sbuild: support of shell exports from
> dpkg_runbuild_prepend
> 
> In the email from Wednesday, 9 March 2022 20:57:32 +03 user Moessbauer,
> Felix
> wrote:
> > Hi,
> >
> > The attached patch adds support to run the sbuild behind a proxy.
> > I briefly tested this with "bitbake mc:qemuamd64-bullseye:isar-image-base"
> > in an environment where proxies are mandatory.
> >
> 
> Hi.
> 
> I've fixed proxy in a bit different way. I found that sbuild supports so called
> 'environment filters' that allow to pass particualr variables to the build
> environment.
> 
> which, besides fixing proxy, also modifies a bit `sbuild_export()` function, so that
> no additional same lines would added to sbuild.conf on the sequential runs of
> build.

I just tested your patch and can confirm that it works.
Probably yours is the better solution anyways, as this has better support for special cases with NO_PROXY, which are hard to handle in the apt config file.

Felix

> 
> I'm not completely sure, but it seems that if some variable is added to
> $environment_filter array, it even doesn't need to be passed via sbuild_export().
> But this should be tested
> 
> > Best regards,
> > Felix
> >
> > > -----Original Message-----
> > > From: isar-users@googlegroups.com <isar-users@googlegroups.com> On
> > > Behalf Of Moessbauer, Felix
> > > Sent: Thursday, March 3, 2022 2:37 PM
> > > To: Uladzimir Bely <ubely@ilbers.de>
> > > Cc: isar-users@googlegroups.com
> > > Subject: RE: [PATCH v6 07/12] sbuild: support of shell exports from
> > > dpkg_runbuild_prepend
> > >
> > > Hi,
> > >
> > > > +
> > > > +        # Don't warn some variables
> > > > +        [ "${var}" = "PARALLEL_MAKE" ] && continue
> > > > +        [ "${var}" = "CCACHE_DIR" ] && continue
> > > > +        [ "${var}" = "PATH_PREPEND" ] && continue
> > > > +        [ "${var}" = "DEB_BUILD_OPTIONS" ] && continue
> > > > +        [ "${var}" = "DEB_BUILD_PROFILES" ] && continue
> > > > +
> > > > +        bbwarn "Export of '${line}' detected, please migrate to
> > > > templates"
> > >
> > > There are more variables that have to be excluded, e.g. all proxy
> > > related
> > > settings:
> > > (http_proxy, HTTP_PROXY, https_proxy, HTTPS_PROXY, no_proxy,
> NO_PROXY).
> > >
> > > But the proxy settings have to be passed into the sbuild environment.
> > > Currently, building behind a proxy fails (in do_dpkg_build):
> > >
> > > +-------------------------------------------------------------------
> > > +------
> > > -----+>
> > > | Update chroot
> > > |     |>
> > > +-------------------------------------------------------------------
> > > +------
> > > -----+ [...]
> > > Err:8
> > > https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdeb
> > >
> .debia%2F&amp;data=04%7C01%7Cfelix.moessbauer%40siemens.com%7C0182
> a5
> > >
> c5e2f243aea7a208da05b627af%7C38ae3bcd95794fd4addab42e1495d55a%7C1
> %7C
> > >
> 0%7C637828577756022287%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjA
> wMDAiL
> > >
> CJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=gUeP
> w
> > > NL3Ypy8%2BJ0pera559edAH48WumjUlIAseLaDOU%3D&amp;reserved=0
> > >
> n.org%2Fdebian&amp;data=04%7C01%7Cfelix.moessbauer%40siemens.com%7
> > >
> C2c6b2941e0eb40ffccb408d9fd1ad566%7C38ae3bcd95794fd4addab42e1495d5
> > >
> 5a%7C1%7C0%7C637819114851078826%7CUnknown%7CTWFpbGZsb3d8eyJWI
> > >
> joiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C300
> > >
> 0&amp;sdata=aLGsi%2BBgf28Cm5g7GGhBY5h3eRu7cL5do5Ern45lsVA%3D&am
> > > p;reserved=0 bullseye Release
> > >
> > >   Cannot initiate the connection to deb.debian.org:80 (2a04:4e42:62::644).
> > >   -
> > >
> > > connect (101: Network is unreachable)
> > >
> > > Best regards,
> > > Felix
> > >
> > > --
> > > Siemens AG, Technology, T CED SES-DE Otto-Hahn-Ring 6, 81739
> > > München, Germany
> > >
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "isar-users" group.
> > > To unsubscribe from this group and stop receiving emails from it,
> > > send an email to isar-users+unsubscribe@googlegroups.com.
> > > To view this discussion on the web visit
> > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgr
> > >
> oups.g%2F&amp;data=04%7C01%7Cfelix.moessbauer%40siemens.com%7C018
> 2a5
> > >
> c5e2f243aea7a208da05b627af%7C38ae3bcd95794fd4addab42e1495d55a%7C1
> %7C
> > >
> 0%7C637828577756022287%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjA
> wMDAiL
> > >
> CJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=82brx
> > > SbpuvOdzp1j1T7e7lx7GhRKZREwKS%2BDTkA0TIw%3D&amp;reserved=0
> > > oogle.com%2Fd%2Fmsgid%2Fisar-
> > >
> users%2FAM9PR10MB48698F892FBA3A33D2C0405289049%2540AM9PR10MB4
> > >
> 869.EURPRD10.PROD.OUTLOOK.COM&amp;data=04%7C01%7Cfelix.moessbau
> > >
> er%40siemens.com%7C2c6b2941e0eb40ffccb408d9fd1ad566%7C38ae3bcd957
> > >
> 94fd4addab42e1495d55a%7C1%7C0%7C637819114851078826%7CUnknown%7
> > >
> CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJ
> > >
> XVCI6Mn0%3D%7C3000&amp;sdata=PGDjMxGm4JkrqZoktZXAJakrj3Xmxt1w8Gq
> > > vVVYmL5w%3D&amp;reserved=0.
> 
> 
> --
> Uladzimir Bely
> 
>