[PATCH v5 00/12] Sbuild/Schroot migration

[PATCH v5 00/12] Sbuild/Schroot migration

[Uladzimir Bely]

This is a patchset showing how sbuild/schroot tools can be integrated
into Isar build system.

This patchset should be applied on top of "preparation" patchset
named "Avoid using shell environment during the build".

Base schroot image is created with sbuild-chroot-target (or
sbuild-chroot-host in case of cross-build) recipe. These images are based
on isar-bootstrap ones, but include some build-related stuff preinstalled.

To use this changes you need to have sbuild and schroot installed and
configured. Current user should be added to sbuild group.

If 'kas-container' is used, it should be modified to support sbuild:
- /var/lib/schroot/union/overlay should be externally mounted (-v option)

Currently, sbuild-related stuff is supported in kas-isar:next image

To use the patchset with KAS:
- export KAS_IMAGE_VERSION=next

If 'gitlab' is used, the similar changes are required:
- `next` tag for kas image should be set
- external dir for schroot overlay should also be specified in
/etc/gitlab-runner/config.toml:
volumes = ["/path/to/overlay:/var/lib/schroot/union/overlay"]

The patchset was tested with 'meta-iot2050' and 'xenomai-images'
downstreams in combination with kas image mentioned above.

Current limitations:
- parallel building of several packages requires more free space in
comparison with buildchroot-based architecture. This happens due
the sbuild architecture which uses some basic layer (common for all
packages) and per-package separate layer (where builddeps are downloaded
and installed).

Changes since v4:
- consider shell exports done in dpkg_runbuild_prepend and pass them
to sbuild environment;
- fixed ccache work.

Changes since v3:
- dpkg_do_mounts() and dpkg_undo_mounts() are not removed for downstreams
compatibility;
- dpkg_build_export is used for adjusting sbuild environment;
- DEB_BUILD_PROFILES support for sbuild is fixed in case of cross-build.
Changes since v2:
- patches reworked/squashed for easier reading and understanding;
- fixed building foreigh architectures with kas-docker;
- implemented support of ccache;
- fixed devshell and devshell_nodeps,
Changes since v1:
- parallel builds with different BUILD_DIR are supported;
- parallel multiconfig targets in one build are supported;
- per-task schroot configuration in /etc/schroot/ is now used;
- patchset now passes Jenkins CI (so patches changes RFC => PATCH).

Anton Mikanovich (1):
  dpkg: Build packages with sbuild

Uladzimir Bely (11):
  dpkg-gbp: Use separate command to export tarball
  dpkg-gbp: Use host tools for dsc preparation
  sbuild: Add recipes for host and target rootfs to run sbuild
  sbuild: Introduce a class for another build method
  sbuild: support of DEB_BUILD_PROFILES
  sbuild: support of shell exports from dpkg_runbuild_prepend
  dpkg: Remove builddeps install task.
  sbuild: add ccache support
  dpkg-base: Switch devshell to use schroot
  dpkg-base: Switch apt_fetch and apt_unpack to use schroot
  doc: Add sbuild-related documentation

 doc/user_manual.md                            |  22 ++-
 meta/classes/dpkg-base.bbclass                |  79 ++++----
 meta/classes/dpkg-gbp.bbclass                 |  26 +--
 meta/classes/dpkg.bbclass                     |  80 +++++---
 meta/classes/sbuild.bbclass                   | 184 ++++++++++++++++++
 meta/conf/bitbake.conf                        |   2 +
 .../sbuild-chroot/sbuild-chroot-host.bb       |  13 ++
 .../sbuild-chroot/sbuild-chroot-target.bb     |  10 +
 .../sbuild-chroot/sbuild-chroot.inc           |  39 ++++
 9 files changed, 375 insertions(+), 80 deletions(-)
 create mode 100644 meta/classes/sbuild.bbclass
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc

-- 
2.20.1

[PATCH v5 01/12] dpkg-gbp: Use separate command to export tarball

[Uladzimir Bely]

We don't actually build the package with gbp, but only prepare it for
the building with external dpkg-buildpackage command. In case there is
no need in real build we can perform only export which will produce
upstream tarball to pass for later building.
This allows to use any build tool for later processing.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-gbp.bbclass | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index d956e8c3..ddf23ca5 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -30,5 +30,9 @@ do_install_builddeps_append() {
 }
 
 dpkg_runbuild_prepend() {
-    export GBP_PREFIX="gbp buildpackage --git-ignore-new ${GBP_EXTRA_OPTIONS} --git-builder="
+    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+        sh -c "cd ${PP}/${PPS} && gbp buildpackage --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}"
+    # NOTE: `buildpackage --git-builder=/bin/true --git-pristine-tar` is used
+    # for compatibility with gbp version froms debian-stretch. In newer distros
+    # it's possible to use a subcommand `export-orig --pristine-tar`
 }
-- 
2.20.1

[PATCH v5 02/12] dpkg-gbp: Use host tools for dsc preparation

[Uladzimir Bely]

Instead of preinstalling gbp-related packages in sbuild chroot
we will use their host versions in order to keep sbuild environment
as clean as possible.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 doc/user_manual.md            |  4 +++-
 meta/classes/dpkg-gbp.bbclass | 24 ++++--------------------
 2 files changed, 7 insertions(+), 21 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 4d4fa763..df73ef89 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -80,7 +80,9 @@ apt install \
   qemu \
   qemu-user-static \
   reprepro \
-  sudo
+  sudo \
+  git-buildpackage \
+  pristine-tar
 ```
 
 If your host is >= buster, also install the following package.
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index ddf23ca5..2fc37b2f 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -9,29 +9,13 @@ S = "${WORKDIR}/git"
 
 PATCHTOOL ?= "git"
 
-GBP_DEPENDS ?= "git-buildpackage pristine-tar"
 GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
 
-do_install_builddeps_append() {
-    dpkg_do_mounts
-    distro="${DISTRO}"
-    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
-       distro="${HOST_DISTRO}"
-    fi
-    deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} \
-        apt-get install -y -o Debug::pkgProblemResolver=yes \
-                        --no-install-recommends --download-only ${GBP_DEPENDS}
-    deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} \
-        apt-get install -y -o Debug::pkgProblemResolver=yes \
-                        --no-install-recommends ${GBP_DEPENDS}
-    dpkg_undo_mounts
-}
-
 dpkg_runbuild_prepend() {
-    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
-        sh -c "cd ${PP}/${PPS} && gbp buildpackage --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}"
+    sh -c "
+        cd ${WORKDIR}/${PPS}
+        gbp buildpackage --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}
+    "
     # NOTE: `buildpackage --git-builder=/bin/true --git-pristine-tar` is used
     # for compatibility with gbp version froms debian-stretch. In newer distros
     # it's possible to use a subcommand `export-orig --pristine-tar`
-- 
2.20.1

[PATCH v5 03/12] sbuild: Add recipes for host and target rootfs to run sbuild

[Uladzimir Bely]

Similar to buildchroot, we need a separate rootfs to be used
for schroot.

It's based on bootstrapped rootfs, but includes several
common build-related packages.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 .../sbuild-chroot/sbuild-chroot-host.bb       | 13 +++++++
 .../sbuild-chroot/sbuild-chroot-target.bb     | 10 ++++++
 .../sbuild-chroot/sbuild-chroot.inc           | 36 +++++++++++++++++++
 3 files changed, 59 insertions(+)
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc

diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
new file mode 100644
index 00000000..aa82846b
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
@@ -0,0 +1,13 @@
+# Root filesystem for packages building
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2021 ilbers GmbH
+
+DESCRIPTION = "Isar sbuild/schroot filesystem for host"
+
+SBUILD_VARIANT = "host"
+
+require sbuild-chroot.inc
+
+ROOTFS_ARCH = "${HOST_ARCH}"
+ROOTFS_DISTRO = "${HOST_DISTRO}"
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
new file mode 100644
index 00000000..d75d783b
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
@@ -0,0 +1,10 @@
+# Root filesystem for packages building
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2021 ilbers GmbH
+
+DESCRIPTION = "Isar sbuild/schroot filesystem for target"
+
+SBUILD_VARIANT = "target"
+
+require sbuild-chroot.inc
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
new file mode 100644
index 00000000..177a8a6d
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -0,0 +1,36 @@
+# Common part for build chroot filesystem.
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2021 ilbers GmbH
+
+LICENSE = "gpl-2.0"
+LIC_FILES_CHKSUM = "file://${LAYERDIR_core}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
+
+PV = "1.0"
+
+inherit rootfs
+
+SBUILD_CHROOT_PREINSTALL_COMMON = " \
+    fakeroot \
+    build-essential \
+    debhelper \
+"
+
+SBUILD_CHROOT_PREINSTALL ?= " \
+    ${SBUILD_CHROOT_PREINSTALL_COMMON} \
+"
+
+SBUILD_CHROOT_DIR = "${WORKDIR}/rootfs"
+ROOTFSDIR = "${SBUILD_CHROOT_DIR}"
+ROOTFS_PACKAGES = "${SBUILD_CHROOT_PREINSTALL}"
+
+# We don't need /etc/apt/sources.list.d/isar-apt.list' while it's handled by sbuild
+ROOTFS_CONFIGURE_COMMAND_remove = "rootfs_configure_isar_apt"
+
+DEPLOY_SCHROOT = "${@d.getVar('SCHROOT_' + d.getVar('SBUILD_VARIANT').upper() + '_DIR')}"
+
+do_sbuildchroot_deploy[dirs] = "${DEPLOY_DIR}/schroot-${SBUILD_VARIANT}"
+do_sbuildchroot_deploy() {
+    ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_SCHROOT}"
+}
+addtask sbuildchroot_deploy before do_build after do_rootfs
-- 
2.20.1

[PATCH v5 04/12] sbuild: Introduce a class for another build method

[Uladzimir Bely]

This also adds mounts for base-apt inside schroot and adds import/export
deb files to/from schroot. So that it becomes possible to run second
 `cached` build from local base-apt repo.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/sbuild.bbclass | 160 ++++++++++++++++++++++++++++++++++++
 meta/conf/bitbake.conf      |   2 +
 2 files changed, 162 insertions(+)
 create mode 100644 meta/classes/sbuild.bbclass

diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
new file mode 100644
index 00000000..e4341a6f
--- /dev/null
+++ b/meta/classes/sbuild.bbclass
@@ -0,0 +1,160 @@
+# This software is a part of ISAR.
+# Copyright (C) 2021 ilbers GmbH
+
+SCHROOT_CONF ?= "/etc/schroot"
+
+SCHROOT_MOUNTS ?= ""
+
+python __anonymous() {
+    import pwd
+    d.setVar('SCHROOT_USER', pwd.getpwuid(os.geteuid()).pw_name)
+    d.setVar('SCHROOT_USER_HOME', pwd.getpwuid(os.geteuid()).pw_dir)
+
+    mode = d.getVar('ISAR_CROSS_COMPILE', True)
+    distro_arch = d.getVar('DISTRO_ARCH')
+    if mode == "0" or d.getVar('HOST_ARCH') ==  distro_arch or \
+       (d.getVar('HOST_DISTRO') == "debian-stretch" and distro_arch == "i386"):
+        d.setVar('SBUILD_HOST_ARCH', distro_arch)
+        d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_TARGET_DIR'))
+        dep = "sbuild-chroot-target:do_build"
+    else:
+        d.setVar('SBUILD_HOST_ARCH', d.getVar('HOST_ARCH'))
+        d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_HOST_DIR'))
+        dep = "sbuild-chroot-host:do_build"
+    d.setVar('SCHROOT_DEP', dep)
+}
+
+SBUILD_CHROOT ?= "${DEBDISTRONAME}-${SCHROOT_USER}-${@os.getpid()}"
+SBUILD_CHROOT_RW ?= "${SBUILD_CHROOT}-rw"
+
+SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}"
+SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}"
+SCHROOT_CONF_FILE_RW ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}-rw"
+
+schroot_create_configs() {
+    sudo -s <<'EOSUDO'
+        set -e
+
+        cat << EOF > "${SCHROOT_CONF_FILE}"
+[${SBUILD_CHROOT}]
+type=directory
+directory=${SCHROOT_DIR}
+profile=${SBUILD_CHROOT}
+users=${SCHROOT_USER}
+groups=root,sbuild
+root-users=${SCHROOT_USER}
+root-groups=root,sbuild
+source-root-users=${SCHROOT_USER}
+source-root-groups=root,sbuild
+union-type=overlay
+preserve-environment=true
+EOF
+
+        cat << EOF > "${SCHROOT_CONF_FILE_RW}"
+[${SBUILD_CHROOT_RW}]
+type=directory
+directory=${SCHROOT_DIR}
+profile=${SBUILD_CHROOT}
+users=${SCHROOT_USER}
+groups=root,sbuild
+root-users=${SCHROOT_USER}
+root-groups=root,sbuild
+preserve-environment=true
+EOF
+
+        mkdir -p "${SCHROOT_DIR}/etc/apt/preferences.d"
+        cat << EOF > "${SCHROOT_DIR}/etc/apt/preferences.d/isar-apt"
+Package: *
+Pin: release n=${DEBDISTRONAME}
+Pin-Priority: 1000
+EOF
+
+        # Prepare mount points
+        cp -rf "${SCHROOT_CONF}/sbuild" "${SBUILD_CONF_DIR}"
+        sbuild_fstab="${SBUILD_CONF_DIR}/fstab"
+
+        fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind 0 0"
+        grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}
+
+        if [ -d ${DL_DIR} ]; then
+            fstab_downloads="${DL_DIR} /downloads none rw,bind 0 0"
+            grep -qxF "${fstab_downloads}" ${sbuild_fstab} || echo "${fstab_downloads}" >> ${sbuild_fstab}
+        fi
+EOSUDO
+}
+
+schroot_delete_configs() {
+    sudo -s <<'EOSUDO'
+        set -e
+        if [ -d "${SBUILD_CONF_DIR}" ]; then
+            rm -rf "${SBUILD_CONF_DIR}"
+        fi
+        rm -f "${SCHROOT_DIR}/etc/apt/preferences.d/isar-apt"
+        rm -f "${SCHROOT_CONF_FILE}"
+        rm -f "${SCHROOT_CONF_FILE_RW}"
+EOSUDO
+}
+
+sbuild_export() {
+    SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
+    VAR=${1}; shift
+    VAR_LINE="'${VAR}' => '${@}',"
+    if [ -s "${SBUILD_CONFIG}" ]; then
+        sed -i -e "\$i\\" -e "${VAR_LINE}" ${SBUILD_CONFIG}
+    else
+        echo "\$build_environment = {" > ${SBUILD_CONFIG}
+        echo "${VAR_LINE}" >> ${SBUILD_CONFIG}
+        echo "};" >> ${SBUILD_CONFIG}
+    fi
+    export SBUILD_CONFIG="${SBUILD_CONFIG}"
+}
+
+schroot_install() {
+    schroot_create_configs
+    APTS="$1"
+
+    distro="${DISTRO}"
+    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+       distro="${HOST_DISTRO}"
+    fi
+
+    deb_dl_dir_import ${SCHROOT_DIR} ${distro}
+    schroot -d / -c ${SBUILD_CHROOT_RW} -u root -- \
+        apt install -y -o Debug::pkgProblemResolver=yes \
+                    --no-install-recommends --download-only ${APTS}
+    deb_dl_dir_export ${SCHROOT_DIR} ${distro}
+
+    schroot -d / -c ${SBUILD_CHROOT_RW} -u root -- \
+        apt install -y -o Debug::pkgProblemResolver=yes \
+                    --no-install-recommends ${APTS}
+    schroot_delete_configs
+}
+
+insert_mounts() {
+    sudo -s <<'EOSUDO'
+        set -e
+        for mp in ${SCHROOT_MOUNTS}; do
+            FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
+            grep -qxF "${FSTAB_LINE}" ${SBUILD_CONF_DIR}/fstab || \
+                echo "${FSTAB_LINE}" >> ${SBUILD_CONF_DIR}/fstab
+        done
+EOSUDO
+}
+
+remove_mounts() {
+    sudo -s <<'EOSUDO'
+        set -e
+        for mp in ${SCHROOT_MOUNTS}; do
+            FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
+            sed -i "\|${FSTAB_LINE}|d" ${SBUILD_CONF_DIR}/fstab
+        done
+EOSUDO
+}
+
+schroot_run() {
+    schroot_create_configs
+    insert_mounts
+    schroot $@
+    remove_mounts
+    schroot_delete_configs
+}
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index cd1c4a64..c1458f05 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -58,6 +58,8 @@ SSTATE_DIR ?= "${TOPDIR}/sstate-cache"
 SSTATE_MANIFESTS = "${TMPDIR}/sstate-control/${DISTRO}-${DISTRO_ARCH}"
 BUILDCHROOT_HOST_DIR = "${DEPLOY_DIR_BUILDCHROOT}-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
 BUILDCHROOT_TARGET_DIR = "${DEPLOY_DIR_BUILDCHROOT}-target/${DISTRO}-${DISTRO_ARCH}"
+SCHROOT_HOST_DIR = "${DEPLOY_DIR}/schroot-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
+SCHROOT_TARGET_DIR = "${DEPLOY_DIR}/schroot-target/${DISTRO}-${DISTRO_ARCH}"
 SDKCHROOT_DIR = "${DEPLOY_DIR_SDKCHROOT}/${DISTRO}-${DISTRO_ARCH}"
 CACHE = "${TMPDIR}/cache"
 KERNEL_FILE ?= "vmlinuz"
-- 
2.20.1

[PATCH v5 05/12] dpkg: Build packages with sbuild

[Uladzimir Bely]

From: Anton Mikanovich <amikan@ilbers.de>

Use previously exported to schroot rootfs to build the package with
sbuild tool. Provide isar-apt as extra repository to be automatically
add to apt sources.

Also added /home/.git-downloads mount while it's used
as git alternates location.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 31 ++++++++++++++++---------------
 meta/classes/dpkg-gbp.bbclass  |  2 ++
 meta/classes/dpkg.bbclass      | 10 ++++++++--
 meta/classes/sbuild.bbclass    |  4 ++--
 4 files changed, 28 insertions(+), 19 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 2add0b22..048a09c0 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -4,6 +4,7 @@
 #
 # SPDX-License-Identifier: MIT
 
+inherit sbuild
 inherit buildchroot
 inherit debianize
 inherit terminal
@@ -14,6 +15,8 @@ DEPENDS ?= ""
 
 DEPENDS_append_riscv64 = "${@' crossbuild-essential-riscv64' if d.getVar('ISAR_CROSS_COMPILE', True) == '1' and d.getVar('PN') != 'crossbuild-essential-riscv64' else ''}"
 
+ISAR_APT_REPO ?= "deb [trusted=yes] file:///home/builder/${PN}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO} ${DEBDISTRONAME} main"
+
 python do_adjust_git() {
     import subprocess
 
@@ -171,6 +174,7 @@ addtask prepare_build after do_patch do_transform_template before do_dpkg_build
 # If Isar recipes depend on each other, they typically need the package
 # deployed to isar-apt
 do_prepare_build[deptask] = "do_deploy_deb"
+do_prepare_build[depends] = "${SCHROOT_DEP}"
 
 BUILDROOT = "${BUILDCHROOT_DIR}/${PP}"
 
@@ -196,20 +200,26 @@ dpkg_undo_mounts() {
     sudo rmdir ${BUILDROOT}
 }
 
+do_prepare_build_append() {
+    # Make a local copy of isar-apt repo that is not affected by other parallel builds
+    mkdir -p ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
+    rm -rf ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/*
+    cp -Rl ${REPO_ISAR_DIR} ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
+}
+
+do_prepare_build[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
+
 # Placeholder for actual dpkg_runbuild() implementation
 dpkg_runbuild() {
     die "This should never be called, overwrite it in your derived class"
 }
 
 python do_dpkg_build() {
-    lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
-                             shared=True)
-    bb.build.exec_func("dpkg_do_mounts", d)
+    bb.build.exec_func('schroot_create_configs', d)
     try:
         bb.build.exec_func("dpkg_runbuild", d)
     finally:
-        bb.build.exec_func("dpkg_undo_mounts", d)
-        bb.utils.unlockfile(lock)
+        bb.build.exec_func('schroot_delete_configs', d)
 }
 
 addtask dpkg_build
@@ -241,7 +251,7 @@ python do_dpkg_build_setscene() {
 addtask dpkg_build_setscene
 do_dpkg_build_setscene[dirs] += "${S}/.."
 
-KEEP_INSTALLED_ON_CLEAN ?= "0"
+do_dpkg_build[depends] = "${SCHROOT_DEP}"
 
 CLEANFUNCS += "deb_clean"
 
@@ -251,15 +261,6 @@ deb_clean() {
         for d in ${DEBS}; do
             repo_del_package "${REPO_ISAR_DIR}"/"${DISTRO}" \
                 "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${d}"
-            if [ "${KEEP_INSTALLED_ON_CLEAN}" = "1" ]; then
-                continue;
-            fi
-            package=$(basename "${d}")
-            package_remove="/usr/bin/apt-get remove -y ${package%%_*}"
-            sudo -E chroot ${BUILDCHROOT_DIR} ${package_remove} || true
-            if [ "${BUILDCHROOT_DIR}" != "${BUILDCHROOT_TARGET_DIR}" ]; then
-                    sudo -E chroot ${BUILDCHROOT_TARGET_DIR} ${package_remove} || true
-            fi
         done
     fi
 }
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index 2fc37b2f..7eda9b03 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -11,6 +11,8 @@ PATCHTOOL ?= "git"
 
 GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
 
+SCHROOT_MOUNTS = "${WORKDIR}:${PP} ${GITDIR}:/home/.git-downloads"
+
 dpkg_runbuild_prepend() {
     sh -c "
         cd ${WORKDIR}/${PPS}
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 27fe84f4..d7afea18 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -34,6 +34,12 @@ dpkg_runbuild() {
     E="${@ isar_export_proxies(d)}"
     E="${@ isar_export_ccache(d)}"
     export PARALLEL_MAKE="${PARALLEL_MAKE}"
-    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
-         /isar/build.sh ${PP}/${PPS} ${PACKAGE_ARCH}
+
+    export SBUILD_CONFIG="${SBUILD_CONFIG}"
+
+    sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
+        --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \
+        --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
+        --debbuildopts="--source-option=-I" \
+        --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
 }
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index e4341a6f..6fff664f 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -31,6 +31,8 @@ SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}"
 SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}"
 SCHROOT_CONF_FILE_RW ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}-rw"
 
+SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
+
 schroot_create_configs() {
     sudo -s <<'EOSUDO'
         set -e
@@ -96,7 +98,6 @@ EOSUDO
 }
 
 sbuild_export() {
-    SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
     VAR=${1}; shift
     VAR_LINE="'${VAR}' => '${@}',"
     if [ -s "${SBUILD_CONFIG}" ]; then
@@ -106,7 +107,6 @@ sbuild_export() {
         echo "${VAR_LINE}" >> ${SBUILD_CONFIG}
         echo "};" >> ${SBUILD_CONFIG}
     fi
-    export SBUILD_CONFIG="${SBUILD_CONFIG}"
 }
 
 schroot_install() {
-- 
2.20.1

[PATCH v5 06/12] sbuild: support of DEB_BUILD_PROFILES

[Uladzimir Bely]

In Isar packages may use their own profiles. For example, linux-custom.inc
introduces 'nolibcdev' profile that should be passed to dpkg-buildpackage
in both cross and native builds.

By default, sbuild for cross-building uses passes "cross,nocheck" options
to dpkg-buildpackage and ignores DEB_BUILD_PROFILES environment variable.

This change makes sbuild use custom profiles even in cross-build mode.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index d7afea18..66db7ec5 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -35,10 +35,18 @@ dpkg_runbuild() {
     E="${@ isar_export_ccache(d)}"
     export PARALLEL_MAKE="${PARALLEL_MAKE}"
 
+    profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 | cut -d "'" -f 4)
+    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+        profiles="${profiles} cross nocheck"
+    fi
+    if [ ! -z "$profiles" ]; then
+        profiles=$(echo --profiles="$profiles" | sed -e 's/ \+/,/g')
+    fi
+
     export SBUILD_CONFIG="${SBUILD_CONFIG}"
 
     sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
-        --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \
+        --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
         --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
         --debbuildopts="--source-option=-I" \
         --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
-- 
2.20.1

[PATCH v5 07/12] sbuild: support of shell exports from dpkg_runbuild_prepend

[Uladzimir Bely]

Many of recipes often use shell exports done in dpkg_run_prepend, so that
this changed environment is used during build.

While sbuild is performed in isolated environment, we need a way to
pass these variables to it. This is done by storing environment before
dpkg_runbuild (after prepare_build) and finding just before the actual
build what was changed or added.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 66db7ec5..c252e9b3 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -29,12 +29,30 @@ do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
 
 addtask devshell after do_install_builddeps
 
+DPKG_PREBUILD_ENV_FILE="${WORKDIR}/dpkg_prebuild.env"
+
+do_prepare_build_append() {
+    env > ${DPKG_PREBUILD_ENV_FILE}
+}
+
 # Build package from sources using build script
 dpkg_runbuild() {
     E="${@ isar_export_proxies(d)}"
     E="${@ isar_export_ccache(d)}"
     export PARALLEL_MAKE="${PARALLEL_MAKE}"
 
+    env | while read -r line; do
+        # Filter the same lines
+        grep -q "^${line}" ${DPKG_PREBUILD_ENV_FILE} && continue
+        # Filter some standard variables
+        echo ${line} | grep -q "^HOME=" && continue
+        echo ${line} | grep -q "^PWD=" && continue
+
+        var=$(echo "${line}" | cut -d '=' -f1)
+        value=$(echo "${line}" | cut -d '=' -f2-)
+        sbuild_export $var "$value"
+    done
+
     profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 | cut -d "'" -f 4)
     if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
         profiles="${profiles} cross nocheck"
-- 
2.20.1

[PATCH v5 08/12] dpkg: Remove builddeps install task.

[Uladzimir Bely]

While builddeps are now handled interlnally by sbuild, we don't need
a task to preinstall them in buildchroot.

But for the second local (base-apt) build we need a way to keep
all dependencies in form of .deb files in DL_DIR. This is done
by executing additional commands in sbuild that copy them to/from
an externally mounted folder.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass | 44 ++++++++++++++++++---------------------
 1 file changed, 20 insertions(+), 24 deletions(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index c252e9b3..8f311deb 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -5,30 +5,6 @@ inherit dpkg-base
 
 PACKAGE_ARCH ?= "${DISTRO_ARCH}"
 
-# Install build dependencies for package
-do_install_builddeps() {
-    dpkg_do_mounts
-    E="${@ isar_export_proxies(d)}"
-    distro="${DISTRO}"
-    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
-       distro="${HOST_DISTRO}"
-    fi
-    deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
-        ${PP}/${PPS} ${PACKAGE_ARCH} --download-only
-    deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
-        ${PP}/${PPS} ${PACKAGE_ARCH}
-    dpkg_undo_mounts
-}
-
-addtask install_builddeps after do_prepare_build before do_dpkg_build
-do_install_builddeps[depends] += "isar-apt:do_cache_config"
-# apt and reprepro may not run in parallel, acquire the Isar lock
-do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
-
-addtask devshell after do_install_builddeps
-
 DPKG_PREBUILD_ENV_FILE="${WORKDIR}/dpkg_prebuild.env"
 
 do_prepare_build_append() {
@@ -53,6 +29,21 @@ dpkg_runbuild() {
         sbuild_export $var "$value"
     done
 
+    distro="${DISTRO}"
+    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+        distro="${HOST_DISTRO}"
+    fi
+
+    deb_dl_dir_import "${WORKDIR}/rootfs" "${distro}"
+
+    deb_dir="/var/cache/apt/archives/"
+    ext_deb_dir="/home/builder/${PN}/rootfs/${deb_dir}"
+
+    ( flock 9
+        grep -qxF '$apt_keep_downloaded_packages = 1;' ${SCHROOT_USER_HOME}/.sbuildrc ||
+            echo '$apt_keep_downloaded_packages = 1;' >> ${SCHROOT_USER_HOME}/.sbuildrc
+    ) 9>"${TMPDIR}/sbuildrc.lock"
+
     profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 | cut -d "'" -f 4)
     if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
         profiles="${profiles} cross nocheck"
@@ -66,6 +57,11 @@ dpkg_runbuild() {
     sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
         --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
         --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
+        --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
+        --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
+        --finished-build-commands="cp -n --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
         --debbuildopts="--source-option=-I" \
         --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
+
+    deb_dl_dir_export "${WORKDIR}/rootfs" "${distro}"
 }
-- 
2.20.1

[PATCH v5 09/12] sbuild: add ccache support

[Uladzimir Bely]

This adds ccache support for custom packages in the same manner
as it was done previously for buildchroot-based build, by using
USE_CCACHE variable in local.conf or per-recipe.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass                     |  4 ++++
 meta/classes/sbuild.bbclass                   | 21 +++++++++++++++++++
 .../sbuild-chroot/sbuild-chroot.inc           |  1 +
 3 files changed, 26 insertions(+)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 8f311deb..ed0e6e35 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -44,6 +44,10 @@ dpkg_runbuild() {
             echo '$apt_keep_downloaded_packages = 1;' >> ${SCHROOT_USER_HOME}/.sbuildrc
     ) 9>"${TMPDIR}/sbuildrc.lock"
 
+    if [ ${USE_CCACHE} -eq 1 ]; then
+        schroot_configure_ccache
+    fi
+
     profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 | cut -d "'" -f 4)
     if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
         profiles="${profiles} cross nocheck"
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 6fff664f..9896b772 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -158,3 +158,24 @@ schroot_run() {
     remove_mounts
     schroot_delete_configs
 }
+
+schroot_configure_ccache() {
+    sudo -s <<'EOSUDO'
+        set -e
+
+        sbuild_fstab="${SBUILD_CONF_DIR}/fstab"
+
+        install --group=sbuild --mode=2775 -d ${CCACHE_DIR}
+        fstab_ccachedir="${CCACHE_DIR} /ccache none rw,bind 0 0"
+        grep -qxF "${fstab_ccachedir}" ${sbuild_fstab} || echo "${fstab_ccachedir}" >> ${sbuild_fstab}
+
+        cat << END > ${CCACHE_DIR}/sbuild-setup
+#!/bin/sh
+export PATH="\$PATH_PREPEND:\$PATH"
+exec "\$@"
+END
+        chmod a+rx ${CCACHE_DIR}/sbuild-setup
+
+        echo "command-prefix=/ccache/sbuild-setup" >> "${SCHROOT_CONF_FILE}"
+EOSUDO
+}
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
index 177a8a6d..74811e22 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -14,6 +14,7 @@ SBUILD_CHROOT_PREINSTALL_COMMON = " \
     fakeroot \
     build-essential \
     debhelper \
+    ccache \
 "
 
 SBUILD_CHROOT_PREINSTALL ?= " \
-- 
2.20.1

[PATCH v5 10/12] dpkg-base: Switch devshell to use schroot

[Uladzimir Bely]

While packages are now build in schroot with sbuild, devshell
task should make terminal open inside sbuild environment.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-base.bbclass                | 23 ++++++++++++++-----
 meta/classes/sbuild.bbclass                   |  3 +++
 .../sbuild-chroot/sbuild-chroot.inc           |  2 ++
 3 files changed, 22 insertions(+), 6 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 048a09c0..b1a16f67 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -278,18 +278,29 @@ do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
 do_deploy_deb[dirs] = "${S}"
 
 python do_devshell() {
-    bb.build.exec_func('dpkg_do_mounts', d)
+    bb.build.exec_func('schroot_create_configs', d)
 
     isar_export_proxies(d)
     isar_export_ccache(d)
 
-    buildchroot = d.getVar('BUILDCHROOT_DIR')
+    schroot = d.getVar('SBUILD_CHROOT')
+    isar_apt = d.getVar('ISAR_APT_REPO')
     pp_pps = os.path.join(d.getVar('PP'), d.getVar('PPS'))
-    # the PATH variable is not forwarded by sudo -E.
-    termcmd = "sudo -E chroot {0} sh -c 'cd {1}; export PATH=$PATH_PREPEND:$PATH; $SHELL -i'"
-    oe_terminal(termcmd.format(buildchroot, pp_pps), "Isar devshell", d)
 
-    bb.build.exec_func('dpkg_undo_mounts', d)
+    install_deps = ":" if d.getVar('BB_CURRENTTASK') == "devshell_nodeps" else "mk-build-deps -i -t \
+        \"apt-get -y -q -o Debug::pkgProblemResolver=yes --no-install-recommends --allow-downgrades\" \
+        debian/control"
+
+    termcmd = "schroot -d / -c {0} -u root -- sh -c ' \
+        cd {1}; \
+        echo {2} > /etc/apt/sources.list.d/isar_apt.list; \
+        apt-get -y -q update; \
+        {3}; \
+        $SHELL -i \
+    '"
+    oe_terminal(termcmd.format(schroot, pp_pps, isar_apt, install_deps), "Isar devshell", d)
+
+    bb.build.exec_func('schroot_delete_configs', d)
 }
 
 addtask devshell after do_prepare_build
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 9896b772..eb358c71 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -78,6 +78,9 @@ EOF
         fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind 0 0"
         grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}
 
+        fstab_pkgdir="${WORKDIR} /home/builder/${PN} none rw,bind 0 0"
+        grep -qxF "${fstab_pkgdir}" ${sbuild_fstab} || echo "${fstab_pkgdir}" >> ${sbuild_fstab}
+
         if [ -d ${DL_DIR} ]; then
             fstab_downloads="${DL_DIR} /downloads none rw,bind 0 0"
             grep -qxF "${fstab_downloads}" ${sbuild_fstab} || echo "${fstab_downloads}" >> ${sbuild_fstab}
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
index 74811e22..4688da1d 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -15,6 +15,8 @@ SBUILD_CHROOT_PREINSTALL_COMMON = " \
     build-essential \
     debhelper \
     ccache \
+    devscripts \
+    equivs \
 "
 
 SBUILD_CHROOT_PREINSTALL ?= " \
-- 
2.20.1

[PATCH v5 11/12] dpkg-base: Switch apt_fetch and apt_unpack to use schroot

[Uladzimir Bely]

This moves downloading and unpacking deb-src to schroot environment.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 25 +++++++++----------------
 1 file changed, 9 insertions(+), 16 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index b1a16f67..39f20263 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -99,19 +99,12 @@ python() {
 }
 
 do_apt_fetch() {
-    dpkg_do_mounts
-    E="${@ isar_export_proxies(d)}"
-    sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update \
-        -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
-        -o Dir::Etc::SourceParts="-" \
-        -o APT::Get::List-Cleanup="0"
-
+    schroot_create_configs
     for uri in "${SRC_APT}"; do
-        sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+        schroot -d / -c ${SBUILD_CHROOT} -- \
             sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only-source source "$2"' my_script "${DISTRO}" "${uri}"
     done
-
-    dpkg_undo_mounts
+    schroot_delete_configs
 }
 
 addtask apt_fetch after do_unpack before do_apt_unpack
@@ -120,13 +113,14 @@ do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
 # Add dependency from the correct buildchroot: host or target
 do_apt_fetch[depends] = "${BUILDCHROOT_DEP}"
 
+# Add dependency from the correct schroot: host or target
+do_apt_fetch[depends] += "${SCHROOT_DEP}"
+
 do_apt_unpack() {
     rm -rf ${S}
-    dpkg_do_mounts
-    E="${@ isar_export_proxies(d)}"
-
+    schroot_create_configs
     for uri in "${SRC_APT}"; do
-        sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+        schroot -d / -c ${SBUILD_CHROOT} -- \
             sh -c ' \
                 set -e
                 dscfile="$(apt-get -y -qq --print-uris --only-source source "${2}" | cut -d " " -f2 | grep -E "*.dsc")"
@@ -135,8 +129,7 @@ do_apt_unpack() {
                 dpkg-source -x "${dscfile}" "${PPS}"' \
                     my_script "${DISTRO}" "${uri}"
     done
-
-    dpkg_undo_mounts
+    schroot_delete_configs
 }
 
 addtask apt_unpack after do_apt_fetch before do_patch
-- 
2.20.1

[PATCH v5 12/12] doc: Add sbuild-related documentation

[Uladzimir Bely]

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 doc/user_manual.md | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index df73ef89..966e7125 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -85,6 +85,24 @@ apt install \
   pristine-tar
 ```
 
+Additional setup is required since `sbuild` is now used for package build.
+Install the following packages:
+```
+apt install \
+  sbuild \
+  schroot
+```
+Also, user who runs isar should be added to `sbuild` group.
+
+**NOTE:** sbuild version (<=0.78.1) packaged in Debian Buster doesn't support
+`$apt_keep_downloaded_packages` option which is required in Isar for
+populating `${DL_DIR}/deb`. So, host `sbuild` in this case should be manually
+upgraded to >=0.81.2 version from Debian Bullseye.
+
+```
+sudo gpasswd -a <username> sbuild
+```
+
 If your host is >= buster, also install the following package.
 ```
 apt install python3-distutils
-- 
2.20.1

Re: [PATCH v5 07/12] sbuild: support of shell exports from dpkg_runbuild_prepend

[Jan Kiszka]

On 01.02.22 18:00, Uladzimir Bely wrote:
> Many of recipes often use shell exports done in dpkg_run_prepend, so that
> this changed environment is used during build.
> 
> While sbuild is performed in isolated environment, we need a way to
> pass these variables to it. This is done by storing environment before
> dpkg_runbuild (after prepare_build) and finding just before the actual
> build what was changed or added.
> 
> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> ---
>  meta/classes/dpkg.bbclass | 18 ++++++++++++++++++
>  1 file changed, 18 insertions(+)
> 
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> index 66db7ec5..c252e9b3 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -29,12 +29,30 @@ do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
>  
>  addtask devshell after do_install_builddeps
>  
> +DPKG_PREBUILD_ENV_FILE="${WORKDIR}/dpkg_prebuild.env"
> +
> +do_prepare_build_append() {
> +    env > ${DPKG_PREBUILD_ENV_FILE}
> +}
> +
>  # Build package from sources using build script
>  dpkg_runbuild() {
>      E="${@ isar_export_proxies(d)}"
>      E="${@ isar_export_ccache(d)}"
>      export PARALLEL_MAKE="${PARALLEL_MAKE}"
>  
> +    env | while read -r line; do
> +        # Filter the same lines
> +        grep -q "^${line}" ${DPKG_PREBUILD_ENV_FILE} && continue
> +        # Filter some standard variables
> +        echo ${line} | grep -q "^HOME=" && continue
> +        echo ${line} | grep -q "^PWD=" && continue
> +
> +        var=$(echo "${line}" | cut -d '=' -f1)
> +        value=$(echo "${line}" | cut -d '=' -f2-)
> +        sbuild_export $var "$value"
> +    done
> +
>      profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 | cut -d "'" -f 4)
>      if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
>          profiles="${profiles} cross nocheck"

So, this basically decouples "Avoid using shell environment during the
build" and similar conversions downstream from this series, right? It's
indeed good to have a compat path now.

However, what are the patterns we want to push? Avoiding exports? Then
we should probably warn here that this compat path should be avoided and
might be removed in the future.

If we want to keep both, we can probably leave several recipes alone in
that other series.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: [PATCH v5 00/12] Sbuild/Schroot migration

[Jan Kiszka]

On 01.02.22 18:00, Uladzimir Bely wrote:
> This is a patchset showing how sbuild/schroot tools can be integrated
> into Isar build system.
> 
> This patchset should be applied on top of "preparation" patchset
> named "Avoid using shell environment during the build".
> 
> Base schroot image is created with sbuild-chroot-target (or
> sbuild-chroot-host in case of cross-build) recipe. These images are based
> on isar-bootstrap ones, but include some build-related stuff preinstalled.
> 
> To use this changes you need to have sbuild and schroot installed and
> configured. Current user should be added to sbuild group.
> 
> If 'kas-container' is used, it should be modified to support sbuild:
> - /var/lib/schroot/union/overlay should be externally mounted (-v option)
> 
> Currently, sbuild-related stuff is supported in kas-isar:next image
> 
> To use the patchset with KAS:
> - export KAS_IMAGE_VERSION=next
> 
> If 'gitlab' is used, the similar changes are required:
> - `next` tag for kas image should be set
> - external dir for schroot overlay should also be specified in
> /etc/gitlab-runner/config.toml:
> volumes = ["/path/to/overlay:/var/lib/schroot/union/overlay"]
> 
> The patchset was tested with 'meta-iot2050' and 'xenomai-images'
> downstreams in combination with kas image mentioned above.
> 
> Current limitations:
> - parallel building of several packages requires more free space in
> comparison with buildchroot-based architecture. This happens due
> the sbuild architecture which uses some basic layer (common for all
> packages) and per-package separate layer (where builddeps are downloaded
> and installed).
> 
> Changes since v4:
> - consider shell exports done in dpkg_runbuild_prepend and pass them
> to sbuild environment;
> - fixed ccache work.
> 
> Changes since v3:
> - dpkg_do_mounts() and dpkg_undo_mounts() are not removed for downstreams
> compatibility;
> - dpkg_build_export is used for adjusting sbuild environment;
> - DEB_BUILD_PROFILES support for sbuild is fixed in case of cross-build.
> Changes since v2:
> - patches reworked/squashed for easier reading and understanding;
> - fixed building foreigh architectures with kas-docker;
> - implemented support of ccache;
> - fixed devshell and devshell_nodeps,
> Changes since v1:
> - parallel builds with different BUILD_DIR are supported;
> - parallel multiconfig targets in one build are supported;
> - per-task schroot configuration in /etc/schroot/ is now used;
> - patchset now passes Jenkins CI (so patches changes RFC => PATCH).
> 
> Anton Mikanovich (1):
>   dpkg: Build packages with sbuild
> 
> Uladzimir Bely (11):
>   dpkg-gbp: Use separate command to export tarball
>   dpkg-gbp: Use host tools for dsc preparation
>   sbuild: Add recipes for host and target rootfs to run sbuild
>   sbuild: Introduce a class for another build method
>   sbuild: support of DEB_BUILD_PROFILES
>   sbuild: support of shell exports from dpkg_runbuild_prepend
>   dpkg: Remove builddeps install task.
>   sbuild: add ccache support
>   dpkg-base: Switch devshell to use schroot
>   dpkg-base: Switch apt_fetch and apt_unpack to use schroot
>   doc: Add sbuild-related documentation
> 
>  doc/user_manual.md                            |  22 ++-
>  meta/classes/dpkg-base.bbclass                |  79 ++++----
>  meta/classes/dpkg-gbp.bbclass                 |  26 +--
>  meta/classes/dpkg.bbclass                     |  80 +++++---
>  meta/classes/sbuild.bbclass                   | 184 ++++++++++++++++++
>  meta/conf/bitbake.conf                        |   2 +
>  .../sbuild-chroot/sbuild-chroot-host.bb       |  13 ++
>  .../sbuild-chroot/sbuild-chroot-target.bb     |  10 +
>  .../sbuild-chroot/sbuild-chroot.inc           |  39 ++++
>  9 files changed, 375 insertions(+), 80 deletions(-)
>  create mode 100644 meta/classes/sbuild.bbclass
>  create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
>  create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
>  create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
> 

Does this functionally obsoletes buildchroot completely? I see that the
recipes are still there. If they aren't used anymore by core Isar, only
kept to make some special, not-yet-converted downstream layers happy,
they may still quickly regress.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: [PATCH v5 00/12] Sbuild/Schroot migration

[Uladzimir Bely]

In the email from Tuesday, 1 February 2022 21:14:12 +03 user Jan Kiszka wrote:
> Does this functionally obsoletes buildchroot completely? I see that the
> recipes are still there. If they aren't used anymore by core Isar, only
> kept to make some special, not-yet-converted downstream layers happy,
> they may still quickly regress.

Not competely. Sbuild is used only for _building_ packages.
Buildchroot is still used, for example, by image.bbclass.

Also, in meta-iot2050 npm.bbclass still uses buildchroot mounts.

Re: [PATCH v5 07/12] sbuild: support of shell exports from dpkg_runbuild_prepend

[Uladzimir Bely]

In the email from Tuesday, 1 February 2022 21:09:07 +03 user Jan Kiszka wrote:
> On 01.02.22 18:00, Uladzimir Bely wrote:
> > Many of recipes often use shell exports done in dpkg_run_prepend, so that
> > this changed environment is used during build.
> > 
> > While sbuild is performed in isolated environment, we need a way to
> > pass these variables to it. This is done by storing environment before
> > dpkg_runbuild (after prepare_build) and finding just before the actual
> > build what was changed or added.
> > 
> > Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> > ---
> > 
> >  meta/classes/dpkg.bbclass | 18 ++++++++++++++++++
> >  1 file changed, 18 insertions(+)
> > 
> > diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> > index 66db7ec5..c252e9b3 100644
> > --- a/meta/classes/dpkg.bbclass
> > +++ b/meta/classes/dpkg.bbclass
> > @@ -29,12 +29,30 @@ do_install_builddeps[lockfiles] +=
> > "${REPO_ISAR_DIR}/isar.lock"> 
> >  addtask devshell after do_install_builddeps
> > 
> > +DPKG_PREBUILD_ENV_FILE="${WORKDIR}/dpkg_prebuild.env"
> > +
> > +do_prepare_build_append() {
> > +    env > ${DPKG_PREBUILD_ENV_FILE}
> > +}
> > +
> > 
> >  # Build package from sources using build script
> >  dpkg_runbuild() {
> >  
> >      E="${@ isar_export_proxies(d)}"
> >      E="${@ isar_export_ccache(d)}"
> >      export PARALLEL_MAKE="${PARALLEL_MAKE}"
> > 
> > +    env | while read -r line; do
> > +        # Filter the same lines
> > +        grep -q "^${line}" ${DPKG_PREBUILD_ENV_FILE} && continue
> > +        # Filter some standard variables
> > +        echo ${line} | grep -q "^HOME=" && continue
> > +        echo ${line} | grep -q "^PWD=" && continue
> > +
> > +        var=$(echo "${line}" | cut -d '=' -f1)
> > +        value=$(echo "${line}" | cut -d '=' -f2-)
> > +        sbuild_export $var "$value"
> > +    done
> > +
> > 
> >      profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 |
> >      cut -d "'" -f 4) if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
> >      
> >          profiles="${profiles} cross nocheck"
> 
> So, this basically decouples "Avoid using shell environment during the
> build" and similar conversions downstream from this series, right? It's
> indeed good to have a compat path now.
> 
> However, what are the patterns we want to push? Avoiding exports? Then
> we should probably warn here that this compat path should be avoided and
> might be removed in the future.
> 
> If we want to keep both, we can probably leave several recipes alone in
> that other series.
> 
> Jan

You might be right. Current solution doesn't force users to avoid using 
exports while they are silently supported.

The first internal solution was keeping dpkg_build_export function but also 
passing custom shell exports (from dpkg_runbuild_prepend) to sbuild 
environment. And a warning about migration was shown.

Later I removed this to make new patchset less invasive for downstreams. 
Anyway, if we agree that "dpkg_build_export" (like API function) is more 
preferable, I can get back this solution with warnings.

Re: [PATCH v5 07/12] sbuild: support of shell exports from dpkg_runbuild_prepend

[Jan Kiszka]

On 01.02.22 19:41, Uladzimir Bely wrote:
> In the email from Tuesday, 1 February 2022 21:09:07 +03 user Jan Kiszka wrote:
>> On 01.02.22 18:00, Uladzimir Bely wrote:
>>> Many of recipes often use shell exports done in dpkg_run_prepend, so that
>>> this changed environment is used during build.
>>>
>>> While sbuild is performed in isolated environment, we need a way to
>>> pass these variables to it. This is done by storing environment before
>>> dpkg_runbuild (after prepare_build) and finding just before the actual
>>> build what was changed or added.
>>>
>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
>>> ---
>>>
>>>  meta/classes/dpkg.bbclass | 18 ++++++++++++++++++
>>>  1 file changed, 18 insertions(+)
>>>
>>> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
>>> index 66db7ec5..c252e9b3 100644
>>> --- a/meta/classes/dpkg.bbclass
>>> +++ b/meta/classes/dpkg.bbclass
>>> @@ -29,12 +29,30 @@ do_install_builddeps[lockfiles] +=
>>> "${REPO_ISAR_DIR}/isar.lock"> 
>>>  addtask devshell after do_install_builddeps
>>>
>>> +DPKG_PREBUILD_ENV_FILE="${WORKDIR}/dpkg_prebuild.env"
>>> +
>>> +do_prepare_build_append() {
>>> +    env > ${DPKG_PREBUILD_ENV_FILE}
>>> +}
>>> +
>>>
>>>  # Build package from sources using build script
>>>  dpkg_runbuild() {
>>>  
>>>      E="${@ isar_export_proxies(d)}"
>>>      E="${@ isar_export_ccache(d)}"
>>>      export PARALLEL_MAKE="${PARALLEL_MAKE}"
>>>
>>> +    env | while read -r line; do
>>> +        # Filter the same lines
>>> +        grep -q "^${line}" ${DPKG_PREBUILD_ENV_FILE} && continue
>>> +        # Filter some standard variables
>>> +        echo ${line} | grep -q "^HOME=" && continue
>>> +        echo ${line} | grep -q "^PWD=" && continue
>>> +
>>> +        var=$(echo "${line}" | cut -d '=' -f1)
>>> +        value=$(echo "${line}" | cut -d '=' -f2-)
>>> +        sbuild_export $var "$value"
>>> +    done
>>> +
>>>
>>>      profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 |
>>>      cut -d "'" -f 4) if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
>>>      
>>>          profiles="${profiles} cross nocheck"
>>
>> So, this basically decouples "Avoid using shell environment during the
>> build" and similar conversions downstream from this series, right? It's
>> indeed good to have a compat path now.
>>
>> However, what are the patterns we want to push? Avoiding exports? Then
>> we should probably warn here that this compat path should be avoided and
>> might be removed in the future.
>>
>> If we want to keep both, we can probably leave several recipes alone in
>> that other series.
>>
>> Jan
> 
> You might be right. Current solution doesn't force users to avoid using 
> exports while they are silently supported.
> 
> The first internal solution was keeping dpkg_build_export function but also 
> passing custom shell exports (from dpkg_runbuild_prepend) to sbuild 
> environment. And a warning about migration was shown.
> 
> Later I removed this to make new patchset less invasive for downstreams. 
> Anyway, if we agree that "dpkg_build_export" (like API function) is more 
> preferable, I can get back this solution with warnings.
> 

This is not necessarily what I'm suggesting. Your conversion series
works without that API, primarily using template. We also have build
profiles. But we first of all need to define what we actually want.

The benefit of pushing things into the build env, rather than feeding it
from the recipe, is likely that the resulting deb-src of such generated
packages can actually be used for rebuilding independently of Isar. So
there is likely value in deprecating classic export.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: [PATCH v5 00/12] Sbuild/Schroot migration

[Jan Kiszka]

On 01.02.22 19:33, Uladzimir Bely wrote:
> In the email from Tuesday, 1 February 2022 21:14:12 +03 user Jan Kiszka wrote:
>> Does this functionally obsoletes buildchroot completely? I see that the
>> recipes are still there. If they aren't used anymore by core Isar, only
>> kept to make some special, not-yet-converted downstream layers happy,
>> they may still quickly regress.
> 
> Not competely. Sbuild is used only for _building_ packages.
> Buildchroot is still used, for example, by image.bbclass.
> 

So, buildchroot becomes "imagechroot" - no longer used for _building_.

> Also, in meta-iot2050 npm.bbclass still uses buildchroot mounts.
> 

Then this needs to be fully converted to sbuild, also regarding the
"install deps via npm" step. That analogous to things like
"linux-custom: Move cfg fragments applying to debian/rules".

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

RE: [PATCH v5 07/12] sbuild: support of shell exports from dpkg_runbuild_prepend

[Moessbauer, Felix]

> -----Original Message-----
> From: isar-users@googlegroups.com <isar-users@googlegroups.com> On
> Behalf Of Jan Kiszka
> Sent: Wednesday, February 2, 2022 7:57 AM
> To: Uladzimir Bely <ubely@ilbers.de>; isar-users@googlegroups.com
> Subject: Re: [PATCH v5 07/12] sbuild: support of shell exports from
> dpkg_runbuild_prepend
> 
> On 01.02.22 19:41, Uladzimir Bely wrote:
> > In the email from Tuesday, 1 February 2022 21:09:07 +03 user Jan Kiszka
> wrote:
> >> On 01.02.22 18:00, Uladzimir Bely wrote:
> >>> Many of recipes often use shell exports done in dpkg_run_prepend, so
> >>> that this changed environment is used during build.
> >>>
> >>> While sbuild is performed in isolated environment, we need a way to
> >>> pass these variables to it. This is done by storing environment
> >>> before dpkg_runbuild (after prepare_build) and finding just before
> >>> the actual build what was changed or added.
> >>>
> >>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> >>> ---
> >>>
> >>>  meta/classes/dpkg.bbclass | 18 ++++++++++++++++++
> >>>  1 file changed, 18 insertions(+)
> >>>
> >>> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> >>> index 66db7ec5..c252e9b3 100644
> >>> --- a/meta/classes/dpkg.bbclass
> >>> +++ b/meta/classes/dpkg.bbclass
> >>> @@ -29,12 +29,30 @@ do_install_builddeps[lockfiles] +=
> >>> "${REPO_ISAR_DIR}/isar.lock">  addtask devshell after
> >>> do_install_builddeps
> >>>
> >>> +DPKG_PREBUILD_ENV_FILE="${WORKDIR}/dpkg_prebuild.env"
> >>> +
> >>> +do_prepare_build_append() {
> >>> +    env > ${DPKG_PREBUILD_ENV_FILE} }
> >>> +
> >>>
> >>>  # Build package from sources using build script
> >>>  dpkg_runbuild() {
> >>>
> >>>      E="${@ isar_export_proxies(d)}"
> >>>      E="${@ isar_export_ccache(d)}"
> >>>      export PARALLEL_MAKE="${PARALLEL_MAKE}"
> >>>
> >>> +    env | while read -r line; do
> >>> +        # Filter the same lines
> >>> +        grep -q "^${line}" ${DPKG_PREBUILD_ENV_FILE} && continue
> >>> +        # Filter some standard variables
> >>> +        echo ${line} | grep -q "^HOME=" && continue
> >>> +        echo ${line} | grep -q "^PWD=" && continue
> >>> +
> >>> +        var=$(echo "${line}" | cut -d '=' -f1)
> >>> +        value=$(echo "${line}" | cut -d '=' -f2-)
> >>> +        sbuild_export $var "$value"
> >>> +    done
> >>> +
> >>>
> >>>      profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 |
> >>>      cut -d "'" -f 4) if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
> >>>
> >>>          profiles="${profiles} cross nocheck"
> >>
> >> So, this basically decouples "Avoid using shell environment during
> >> the build" and similar conversions downstream from this series,
> >> right? It's indeed good to have a compat path now.
> >>
> >> However, what are the patterns we want to push? Avoiding exports?
> >> Then we should probably warn here that this compat path should be
> >> avoided and might be removed in the future.
> >>
> >> If we want to keep both, we can probably leave several recipes alone
> >> in that other series.
> >>
> >> Jan
> >
> > You might be right. Current solution doesn't force users to avoid
> > using exports while they are silently supported.
> >
> > The first internal solution was keeping dpkg_build_export function but
> > also passing custom shell exports (from dpkg_runbuild_prepend) to
> > sbuild environment. And a warning about migration was shown.
> >
> > Later I removed this to make new patchset less invasive for downstreams.
> > Anyway, if we agree that "dpkg_build_export" (like API function) is
> > more preferable, I can get back this solution with warnings.
> >
> 
> This is not necessarily what I'm suggesting. Your conversion series works
> without that API, primarily using template. We also have build profiles. But we
> first of all need to define what we actually want.
> 
> The benefit of pushing things into the build env, rather than feeding it from the
> recipe, is likely that the resulting deb-src of such generated packages can
> actually be used for rebuilding independently of Isar. So there is likely value in
> deprecating classic export.

Using environment variables to control the build in Debian/rules is generally discouraged (except for DEB_BUILD_PROFILES and DEB_BUILD_OPTIONS).
But even for these variables, semantics are documented [1] and we / an ISAR user should strictly follow them when packaging.

There are also reasons why you want to build a package in a clean environment (e.g. to avoid accidentally setting CC or CXX to icc or clang).
Hence, I highly vote for not forwarding the environment into the sbuild build environment.

Felix

[1] https://wiki.debian.org/BuildProfileSpec#Registered_profile_names

> 
> Jan
> 
> --
> Siemens AG, Technology
> Competence Center Embedded Linux
> 
> --
> You received this message because you are subscribed to the Google Groups
> "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to isar-users+unsubscribe@googlegroups.com.
> To view this discussion on the web visit
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.g
> oogle.com%2Fd%2Fmsgid%2Fisar-users%2F9f8b4874-62b5-546f-5dc9-
> 8d9c0bd5b1cf%2540siemens.com&amp;data=04%7C01%7Cfelix.moessbauer%4
> 0siemens.com%7C9ed643fe46b945b7d2a008d9e619471f%7C38ae3bcd95794fd
> 4addab42e1495d55a%7C1%7C0%7C637793819407336907%7CUnknown%7CTW
> FpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI
> 6Mn0%3D%7C3000&amp;sdata=Wfb%2F4NfV1M9LjodozJ1V5p3riY8HHobGS31
> %2B6gJ7y6c%3D&amp;reserved=0.

Re: [PATCH v5 07/12] sbuild: support of shell exports from dpkg_runbuild_prepend

[Jan Kiszka]

On 02.02.22 09:46, Moessbauer, Felix (T CED SES-DE) wrote:
> 
>> -----Original Message-----
>> From: isar-users@googlegroups.com <isar-users@googlegroups.com> On
>> Behalf Of Jan Kiszka
>> Sent: Wednesday, February 2, 2022 7:57 AM
>> To: Uladzimir Bely <ubely@ilbers.de>; isar-users@googlegroups.com
>> Subject: Re: [PATCH v5 07/12] sbuild: support of shell exports from
>> dpkg_runbuild_prepend
>>
>> On 01.02.22 19:41, Uladzimir Bely wrote:
>>> In the email from Tuesday, 1 February 2022 21:09:07 +03 user Jan Kiszka
>> wrote:
>>>> On 01.02.22 18:00, Uladzimir Bely wrote:
>>>>> Many of recipes often use shell exports done in dpkg_run_prepend, so
>>>>> that this changed environment is used during build.
>>>>>
>>>>> While sbuild is performed in isolated environment, we need a way to
>>>>> pass these variables to it. This is done by storing environment
>>>>> before dpkg_runbuild (after prepare_build) and finding just before
>>>>> the actual build what was changed or added.
>>>>>
>>>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
>>>>> ---
>>>>>
>>>>>  meta/classes/dpkg.bbclass | 18 ++++++++++++++++++
>>>>>  1 file changed, 18 insertions(+)
>>>>>
>>>>> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
>>>>> index 66db7ec5..c252e9b3 100644
>>>>> --- a/meta/classes/dpkg.bbclass
>>>>> +++ b/meta/classes/dpkg.bbclass
>>>>> @@ -29,12 +29,30 @@ do_install_builddeps[lockfiles] +=
>>>>> "${REPO_ISAR_DIR}/isar.lock">  addtask devshell after
>>>>> do_install_builddeps
>>>>>
>>>>> +DPKG_PREBUILD_ENV_FILE="${WORKDIR}/dpkg_prebuild.env"
>>>>> +
>>>>> +do_prepare_build_append() {
>>>>> +    env > ${DPKG_PREBUILD_ENV_FILE} }
>>>>> +
>>>>>
>>>>>  # Build package from sources using build script
>>>>>  dpkg_runbuild() {
>>>>>
>>>>>      E="${@ isar_export_proxies(d)}"
>>>>>      E="${@ isar_export_ccache(d)}"
>>>>>      export PARALLEL_MAKE="${PARALLEL_MAKE}"
>>>>>
>>>>> +    env | while read -r line; do
>>>>> +        # Filter the same lines
>>>>> +        grep -q "^${line}" ${DPKG_PREBUILD_ENV_FILE} && continue
>>>>> +        # Filter some standard variables
>>>>> +        echo ${line} | grep -q "^HOME=" && continue
>>>>> +        echo ${line} | grep -q "^PWD=" && continue
>>>>> +
>>>>> +        var=$(echo "${line}" | cut -d '=' -f1)
>>>>> +        value=$(echo "${line}" | cut -d '=' -f2-)
>>>>> +        sbuild_export $var "$value"
>>>>> +    done
>>>>> +
>>>>>
>>>>>      profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 |
>>>>>      cut -d "'" -f 4) if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
>>>>>
>>>>>          profiles="${profiles} cross nocheck"
>>>>
>>>> So, this basically decouples "Avoid using shell environment during
>>>> the build" and similar conversions downstream from this series,
>>>> right? It's indeed good to have a compat path now.
>>>>
>>>> However, what are the patterns we want to push? Avoiding exports?
>>>> Then we should probably warn here that this compat path should be
>>>> avoided and might be removed in the future.
>>>>
>>>> If we want to keep both, we can probably leave several recipes alone
>>>> in that other series.
>>>>
>>>> Jan
>>>
>>> You might be right. Current solution doesn't force users to avoid
>>> using exports while they are silently supported.
>>>
>>> The first internal solution was keeping dpkg_build_export function but
>>> also passing custom shell exports (from dpkg_runbuild_prepend) to
>>> sbuild environment. And a warning about migration was shown.
>>>
>>> Later I removed this to make new patchset less invasive for downstreams.
>>> Anyway, if we agree that "dpkg_build_export" (like API function) is
>>> more preferable, I can get back this solution with warnings.
>>>
>>
>> This is not necessarily what I'm suggesting. Your conversion series works
>> without that API, primarily using template. We also have build profiles. But we
>> first of all need to define what we actually want.
>>
>> The benefit of pushing things into the build env, rather than feeding it from the
>> recipe, is likely that the resulting deb-src of such generated packages can
>> actually be used for rebuilding independently of Isar. So there is likely value in
>> deprecating classic export.
> 
> Using environment variables to control the build in Debian/rules is generally discouraged (except for DEB_BUILD_PROFILES and DEB_BUILD_OPTIONS).
> But even for these variables, semantics are documented [1] and we / an ISAR user should strictly follow them when packaging.
> 
> There are also reasons why you want to build a package in a clean environment (e.g. to avoid accidentally setting CC or CXX to icc or clang).
> Hence, I highly vote for not forwarding the environment into the sbuild build environment.
> 

...without a warning about a grace period during that we will still do
that to allow downstream migration. Then we have a plan, I would say.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: [PATCH v5 07/12] sbuild: support of shell exports from dpkg_runbuild_prepend

[Uladzimir Bely]

In the email from Wednesday, 2 February 2022 11:52:29 +03 user Jan Kiszka 
wrote:
> On 02.02.22 09:46, Moessbauer, Felix (T CED SES-DE) wrote:
> >> -----Original Message-----
> >> From: isar-users@googlegroups.com <isar-users@googlegroups.com> On
> >> Behalf Of Jan Kiszka
> >> Sent: Wednesday, February 2, 2022 7:57 AM
> >> To: Uladzimir Bely <ubely@ilbers.de>; isar-users@googlegroups.com
> >> Subject: Re: [PATCH v5 07/12] sbuild: support of shell exports from
> >> dpkg_runbuild_prepend
> >> 
> >> On 01.02.22 19:41, Uladzimir Bely wrote:
> >>> In the email from Tuesday, 1 February 2022 21:09:07 +03 user Jan Kiszka
> >> 
> >> wrote:
> >>>> On 01.02.22 18:00, Uladzimir Bely wrote:
> >>>>> Many of recipes often use shell exports done in dpkg_run_prepend, so
> >>>>> that this changed environment is used during build.
> >>>>> 
> >>>>> While sbuild is performed in isolated environment, we need a way to
> >>>>> pass these variables to it. This is done by storing environment
> >>>>> before dpkg_runbuild (after prepare_build) and finding just before
> >>>>> the actual build what was changed or added.
> >>>>> 
> >>>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> >>>>> ---
> >>>>> 
> >>>>>  meta/classes/dpkg.bbclass | 18 ++++++++++++++++++
> >>>>>  1 file changed, 18 insertions(+)
> >>>>> 
> >>>>> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> >>>>> index 66db7ec5..c252e9b3 100644
> >>>>> --- a/meta/classes/dpkg.bbclass
> >>>>> +++ b/meta/classes/dpkg.bbclass
> >>>>> @@ -29,12 +29,30 @@ do_install_builddeps[lockfiles] +=
> >>>>> "${REPO_ISAR_DIR}/isar.lock">  addtask devshell after
> >>>>> do_install_builddeps
> >>>>> 
> >>>>> +DPKG_PREBUILD_ENV_FILE="${WORKDIR}/dpkg_prebuild.env"
> >>>>> +
> >>>>> +do_prepare_build_append() {
> >>>>> +    env > ${DPKG_PREBUILD_ENV_FILE} }
> >>>>> +
> >>>>> 
> >>>>>  # Build package from sources using build script
> >>>>>  dpkg_runbuild() {
> >>>>>  
> >>>>>      E="${@ isar_export_proxies(d)}"
> >>>>>      E="${@ isar_export_ccache(d)}"
> >>>>>      export PARALLEL_MAKE="${PARALLEL_MAKE}"
> >>>>> 
> >>>>> +    env | while read -r line; do
> >>>>> +        # Filter the same lines
> >>>>> +        grep -q "^${line}" ${DPKG_PREBUILD_ENV_FILE} && continue
> >>>>> +        # Filter some standard variables
> >>>>> +        echo ${line} | grep -q "^HOME=" && continue
> >>>>> +        echo ${line} | grep -q "^PWD=" && continue
> >>>>> +
> >>>>> +        var=$(echo "${line}" | cut -d '=' -f1)
> >>>>> +        value=$(echo "${line}" | cut -d '=' -f2-)
> >>>>> +        sbuild_export $var "$value"
> >>>>> +    done
> >>>>> +
> >>>>> 
> >>>>>      profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1
> >>>>>      |
> >>>>>      cut -d "'" -f 4) if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
> >>>>>      
> >>>>>          profiles="${profiles} cross nocheck"
> >>>> 
> >>>> So, this basically decouples "Avoid using shell environment during
> >>>> the build" and similar conversions downstream from this series,
> >>>> right? It's indeed good to have a compat path now.
> >>>> 
> >>>> However, what are the patterns we want to push? Avoiding exports?
> >>>> Then we should probably warn here that this compat path should be
> >>>> avoided and might be removed in the future.
> >>>> 
> >>>> If we want to keep both, we can probably leave several recipes alone
> >>>> in that other series.
> >>>> 
> >>>> Jan
> >>> 
> >>> You might be right. Current solution doesn't force users to avoid
> >>> using exports while they are silently supported.
> >>> 
> >>> The first internal solution was keeping dpkg_build_export function but
> >>> also passing custom shell exports (from dpkg_runbuild_prepend) to
> >>> sbuild environment. And a warning about migration was shown.
> >>> 
> >>> Later I removed this to make new patchset less invasive for downstreams.
> >>> Anyway, if we agree that "dpkg_build_export" (like API function) is
> >>> more preferable, I can get back this solution with warnings.
> >> 
> >> This is not necessarily what I'm suggesting. Your conversion series works
> >> without that API, primarily using template. We also have build profiles.
> >> But we first of all need to define what we actually want.
> >> 
> >> The benefit of pushing things into the build env, rather than feeding it
> >> from the recipe, is likely that the resulting deb-src of such generated
> >> packages can actually be used for rebuilding independently of Isar. So
> >> there is likely value in deprecating classic export.
> > 
> > Using environment variables to control the build in Debian/rules is
> > generally discouraged (except for DEB_BUILD_PROFILES and
> > DEB_BUILD_OPTIONS). But even for these variables, semantics are
> > documented [1] and we / an ISAR user should strictly follow them when
> > packaging.
> > 
> > There are also reasons why you want to build a package in a clean
> > environment (e.g. to avoid accidentally setting CC or CXX to icc or
> > clang). Hence, I highly vote for not forwarding the environment into the
> > sbuild build environment.
> ...without a warning about a grace period during that we will still do
> that to allow downstream migration. Then we have a plan, I would say.
> 
> Jan

To make things clear, there are now three options:

1) (Current) Just support passing shell exports user done in 
dpkg_runbuild_prepend to (s)build environment. Users of downstreams are not 
notified, so they continue using these exports. 

2) Continue support custom shell exports by passing them to (s)build env, BUT 
show warnings, that user should use some 'dpkg_build_export'. In general, this 
doesn't differ much from the previous approach.

3) Continue support custom shell exports by passing them to (s)build env, BUT 
show warnings, that this stuff should be converted to templates. After some 
period, stop supporting exports.

4) Doesn't support shell exports passing to (s)build env, so it will 
immediately break the builds.

As I understand correctly, the way (3) is the best solution. 

-- 
Uladzimir Bely

Re: [PATCH v5 07/12] sbuild: support of shell exports from dpkg_runbuild_prepend

[Jan Kiszka]

On 02.02.22 10:13, Uladzimir Bely wrote:
> In the email from Wednesday, 2 February 2022 11:52:29 +03 user Jan Kiszka 
> wrote:
>> On 02.02.22 09:46, Moessbauer, Felix (T CED SES-DE) wrote:
>>>> -----Original Message-----
>>>> From: isar-users@googlegroups.com <isar-users@googlegroups.com> On
>>>> Behalf Of Jan Kiszka
>>>> Sent: Wednesday, February 2, 2022 7:57 AM
>>>> To: Uladzimir Bely <ubely@ilbers.de>; isar-users@googlegroups.com
>>>> Subject: Re: [PATCH v5 07/12] sbuild: support of shell exports from
>>>> dpkg_runbuild_prepend
>>>>
>>>> On 01.02.22 19:41, Uladzimir Bely wrote:
>>>>> In the email from Tuesday, 1 February 2022 21:09:07 +03 user Jan Kiszka
>>>>
>>>> wrote:
>>>>>> On 01.02.22 18:00, Uladzimir Bely wrote:
>>>>>>> Many of recipes often use shell exports done in dpkg_run_prepend, so
>>>>>>> that this changed environment is used during build.
>>>>>>>
>>>>>>> While sbuild is performed in isolated environment, we need a way to
>>>>>>> pass these variables to it. This is done by storing environment
>>>>>>> before dpkg_runbuild (after prepare_build) and finding just before
>>>>>>> the actual build what was changed or added.
>>>>>>>
>>>>>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
>>>>>>> ---
>>>>>>>
>>>>>>>  meta/classes/dpkg.bbclass | 18 ++++++++++++++++++
>>>>>>>  1 file changed, 18 insertions(+)
>>>>>>>
>>>>>>> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
>>>>>>> index 66db7ec5..c252e9b3 100644
>>>>>>> --- a/meta/classes/dpkg.bbclass
>>>>>>> +++ b/meta/classes/dpkg.bbclass
>>>>>>> @@ -29,12 +29,30 @@ do_install_builddeps[lockfiles] +=
>>>>>>> "${REPO_ISAR_DIR}/isar.lock">  addtask devshell after
>>>>>>> do_install_builddeps
>>>>>>>
>>>>>>> +DPKG_PREBUILD_ENV_FILE="${WORKDIR}/dpkg_prebuild.env"
>>>>>>> +
>>>>>>> +do_prepare_build_append() {
>>>>>>> +    env > ${DPKG_PREBUILD_ENV_FILE} }
>>>>>>> +
>>>>>>>
>>>>>>>  # Build package from sources using build script
>>>>>>>  dpkg_runbuild() {
>>>>>>>  
>>>>>>>      E="${@ isar_export_proxies(d)}"
>>>>>>>      E="${@ isar_export_ccache(d)}"
>>>>>>>      export PARALLEL_MAKE="${PARALLEL_MAKE}"
>>>>>>>
>>>>>>> +    env | while read -r line; do
>>>>>>> +        # Filter the same lines
>>>>>>> +        grep -q "^${line}" ${DPKG_PREBUILD_ENV_FILE} && continue
>>>>>>> +        # Filter some standard variables
>>>>>>> +        echo ${line} | grep -q "^HOME=" && continue
>>>>>>> +        echo ${line} | grep -q "^PWD=" && continue
>>>>>>> +
>>>>>>> +        var=$(echo "${line}" | cut -d '=' -f1)
>>>>>>> +        value=$(echo "${line}" | cut -d '=' -f2-)
>>>>>>> +        sbuild_export $var "$value"
>>>>>>> +    done
>>>>>>> +
>>>>>>>
>>>>>>>      profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1
>>>>>>>      |
>>>>>>>      cut -d "'" -f 4) if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
>>>>>>>      
>>>>>>>          profiles="${profiles} cross nocheck"
>>>>>>
>>>>>> So, this basically decouples "Avoid using shell environment during
>>>>>> the build" and similar conversions downstream from this series,
>>>>>> right? It's indeed good to have a compat path now.
>>>>>>
>>>>>> However, what are the patterns we want to push? Avoiding exports?
>>>>>> Then we should probably warn here that this compat path should be
>>>>>> avoided and might be removed in the future.
>>>>>>
>>>>>> If we want to keep both, we can probably leave several recipes alone
>>>>>> in that other series.
>>>>>>
>>>>>> Jan
>>>>>
>>>>> You might be right. Current solution doesn't force users to avoid
>>>>> using exports while they are silently supported.
>>>>>
>>>>> The first internal solution was keeping dpkg_build_export function but
>>>>> also passing custom shell exports (from dpkg_runbuild_prepend) to
>>>>> sbuild environment. And a warning about migration was shown.
>>>>>
>>>>> Later I removed this to make new patchset less invasive for downstreams.
>>>>> Anyway, if we agree that "dpkg_build_export" (like API function) is
>>>>> more preferable, I can get back this solution with warnings.
>>>>
>>>> This is not necessarily what I'm suggesting. Your conversion series works
>>>> without that API, primarily using template. We also have build profiles.
>>>> But we first of all need to define what we actually want.
>>>>
>>>> The benefit of pushing things into the build env, rather than feeding it
>>>> from the recipe, is likely that the resulting deb-src of such generated
>>>> packages can actually be used for rebuilding independently of Isar. So
>>>> there is likely value in deprecating classic export.
>>>
>>> Using environment variables to control the build in Debian/rules is
>>> generally discouraged (except for DEB_BUILD_PROFILES and
>>> DEB_BUILD_OPTIONS). But even for these variables, semantics are
>>> documented [1] and we / an ISAR user should strictly follow them when
>>> packaging.
>>>
>>> There are also reasons why you want to build a package in a clean
>>> environment (e.g. to avoid accidentally setting CC or CXX to icc or
>>> clang). Hence, I highly vote for not forwarding the environment into the
>>> sbuild build environment.
>> ...without a warning about a grace period during that we will still do
>> that to allow downstream migration. Then we have a plan, I would say.
>>
>> Jan
> 
> To make things clear, there are now three options:
> 
> 1) (Current) Just support passing shell exports user done in 
> dpkg_runbuild_prepend to (s)build environment. Users of downstreams are not 
> notified, so they continue using these exports. 
> 
> 2) Continue support custom shell exports by passing them to (s)build env, BUT 
> show warnings, that user should use some 'dpkg_build_export'. In general, this 
> doesn't differ much from the previous approach.
> 
> 3) Continue support custom shell exports by passing them to (s)build env, BUT 
> show warnings, that this stuff should be converted to templates. After some 
> period, stop supporting exports.

Templates or build profiles.

> 
> 4) Doesn't support shell exports passing to (s)build env, so it will 
> immediately break the builds.
> 
> As I understand correctly, the way (3) is the best solution. 
> 

That is also my understanding.

Thanks,
Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux