[RFC 00/15] Sbuild/Schroot migration

[RFC 00/15] Sbuild/Schroot migration

[Anton Mikanovich]

This is temporary patchset showing how sbuild/schroot tools can be
integrated into Isar build system. It is not jet finished functionaly
and architectually, also it do not cover all the targets.
Currently isar-bootstrap is exported as base schroot image. Buildchroot
is also used somewhere, removing it still in progress. There are also
several restrictions, like not fully supported parallel build.
To use this changes you need to have sbuild and schroot installed and
configured. Current user should be added to sbuild group.
At least qemuarm-buster target can be build for testing.

TODO:
- Separate schroot prepare from isar-bootstrap.
- Remove buildchroot usage.
- Finish chroot removing.
- Add documentation updates.

Anton Mikanovich (15):
  dpkg: Install raw package files to source root
  dpkg-gbp: Use separate export-orig command
  isar-bootstrap: Export bootstrap to schroot config
  linux-module: Do not use shell environment
  u-boot: Do not use shell environment
  trusted-firmware: Do not use shell environment
  optee-os: Do not use shell environment
  kselftest: Do not use shell environment
  dpkg: Build packages with sbuild
  sbuild: Introduce environment variables export API
  dpkg: Remove builddeps install task
  dpkg-gbp: Migrate to schroot
  img: Use schroot for image prepare
  linux-mainline: Move cfg fragment test to debian/rules
  linux-custom: Prepare kernel config inside sbuild

 .../optee-os/optee-os-stm32mp15x_3.11.0.bb    |  2 +-
 .../linux/linux-mainline_5.4.70.bb            | 16 ++--
 meta/classes/dpkg-gbp.bbclass                 | 19 ++---
 meta/classes/dpkg-raw.bbclass                 |  8 +-
 meta/classes/dpkg.bbclass                     | 38 +++------
 meta/classes/ext4-img.bbclass                 |  8 +-
 meta/classes/fit-img.bbclass                  | 11 +--
 meta/classes/image-tools-extension.bbclass    | 14 +---
 meta/classes/image.bbclass                    | 19 +----
 meta/classes/sbuild.bbclass                   | 78 +++++++++++++++++++
 meta/classes/ubi-img.bbclass                  | 11 +--
 meta/classes/ubifs-img.bbclass                | 11 +--
 .../files/debian/{rules => rules.tmpl}        |  4 +-
 meta/recipes-bsp/optee-os/optee-os-custom.inc | 10 +--
 .../files/debian/{rules => rules.tmpl}        |  4 +-
 .../trusted-firmware-a-custom.inc             | 10 +--
 .../u-boot/files/debian/{rules => rules.tmpl} |  2 +-
 meta/recipes-bsp/u-boot/u-boot-custom.inc     | 10 +--
 .../isar-bootstrap/isar-bootstrap.inc         | 54 ++++++++++++-
 .../kselftest/files/{rules => rules.tmpl}     |  0
 meta/recipes-kernel/kselftest/kselftest.inc   | 21 ++---
 .../files/debian/{rules => rules.tmpl}        | 10 ++-
 meta/recipes-kernel/linux-module/module.inc   | 23 ++----
 meta/recipes-kernel/linux/linux-custom.inc    |  7 +-
 24 files changed, 225 insertions(+), 165 deletions(-)
 create mode 100644 meta/classes/sbuild.bbclass
 rename meta/recipes-bsp/optee-os/files/debian/{rules => rules.tmpl} (75%)
 rename meta/recipes-bsp/trusted-firmware-a/files/debian/{rules => rules.tmpl} (77%)
 rename meta/recipes-bsp/u-boot/files/debian/{rules => rules.tmpl} (93%)
 rename meta/recipes-kernel/kselftest/files/{rules => rules.tmpl} (100%)
 rename meta/recipes-kernel/linux-module/files/debian/{rules => rules.tmpl} (55%)

-- 
2.25.1

[RFC 01/15] dpkg: Install raw package files to source root

[Anton Mikanovich]

To get rid of using absolute paths in debian/rules move D to source
root. This is default location for dh_install, so now there is no need
in mention it in debian/rules.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/dpkg-raw.bbclass | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)

diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
index dd7b761..73356fa 100644
--- a/meta/classes/dpkg-raw.bbclass
+++ b/meta/classes/dpkg-raw.bbclass
@@ -5,7 +5,7 @@
 
 inherit dpkg
 
-D = "${WORKDIR}/image"
+D = "${S}"
 
 # Populate folder that will be picked up as package
 do_install() {
@@ -22,10 +22,4 @@ do_prepare_build() {
 		sed 's:^./::' > ${S}/debian/${PN}.install
 
 	deb_debianize
-
-	cat <<EOF >> ${S}/debian/rules
-
-override_dh_install:
-	dh_install --sourcedir=${PP}/image
-EOF
 }
-- 
2.25.1

[RFC 02/15] dpkg-gbp: Use separate export-orig command

[Anton Mikanovich]

We don't actually build the package with gbp, but only prepare it for
the building with external dpkg-buildpackage command. In case there is
no need in real build we can perform only export-orig command which
will produce upstream tarball to pass for later building.
This allows to use any build tool for later processing.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/dpkg-gbp.bbclass | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index 20d2d4c..2711e6c 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -10,7 +10,7 @@ S = "${WORKDIR}/git"
 PATCHTOOL ?= "git"
 
 GBP_DEPENDS ?= "git-buildpackage pristine-tar"
-GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
+GBP_EXTRA_OPTIONS ?= "--pristine-tar"
 
 builddeps_install_append() {
     deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
@@ -24,5 +24,6 @@ builddeps_install_append() {
 }
 
 dpkg_runbuild_prepend() {
-    export GBP_PREFIX="gbp buildpackage --git-ignore-new ${GBP_EXTRA_OPTIONS} --git-builder="
+    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+        sh -c "cd ${PP}/${PPS} && gbp export-orig ${GBP_EXTRA_OPTIONS}"
 }
-- 
2.25.1

[RFC 03/15] isar-bootstrap: Export bootstrap to schroot config

[Anton Mikanovich]

To use isar-bootstrap with sbuild and schroot tools we need to export
the path to schroot config.

We also need to put isar-apt and downloads into fstab for sbuild
profile to have it mounted inside schroot.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/sbuild.bbclass                   | 17 ++++++
 .../isar-bootstrap/isar-bootstrap.inc         | 54 ++++++++++++++++++-
 2 files changed, 69 insertions(+), 2 deletions(-)
 create mode 100644 meta/classes/sbuild.bbclass

diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
new file mode 100644
index 0000000..ed2d0cf
--- /dev/null
+++ b/meta/classes/sbuild.bbclass
@@ -0,0 +1,17 @@
+# This software is a part of ISAR.
+# Copyright (C) 2021 ilbers GmbH
+
+SCHROOT_CONF ?= "/etc/schroot"
+
+python __anonymous() {
+    mode = d.getVar('ISAR_CROSS_COMPILE', True)
+    distro_arch = d.getVar('DISTRO_ARCH')
+    if mode == "0" or d.getVar('HOST_ARCH') ==  distro_arch or \
+       (d.getVar('HOST_DISTRO') == "debian-stretch" and distro_arch == "i386"):
+        d.setVar('SBUILD_HOST_ARCH', distro_arch)
+    else:
+        d.setVar('SBUILD_HOST_ARCH', d.getVar('HOST_ARCH'))
+}
+
+SBUILD_CHROOT ?= "${DEBDISTRONAME}-${@os.getlogin()}-${DISTRO}-${SBUILD_HOST_ARCH}"
+SBUILD_CHROOT_RW ?= "${DEBDISTRONAME}-${@os.getlogin()}-${DISTRO}-${SBUILD_HOST_ARCH}-rw"
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 0edefc5..b4f352b 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -23,11 +23,12 @@ APTSRCS_INIT = "${WORKDIR}/apt-sources-init"
 DISTRO_BOOTSTRAP_KEYFILES = ""
 THIRD_PARTY_APT_KEYFILES = ""
 DEPLOY_ISAR_BOOTSTRAP ?= ""
-DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"
+DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales,fakeroot,build-essential,debhelper"
 DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg"
 DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = "${@https_support(d)}"
 
 inherit deb-dl-dir
+inherit sbuild
 
 python () {
     distro_bootstrap_keys = (d.getVar("DISTRO_BOOTSTRAP_KEYS") or "").split()
@@ -254,8 +255,9 @@ isar_bootstrap() {
     if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" -a -z "${BASE_REPO_KEY}" ]; then
         debootstrap_args="$debootstrap_args --no-check-gpg"
     fi
+    user_name=$(id -un)
     E="${@ isar_export_proxies(d)}"
-    export IS_HOST debootstrap_args E
+    export IS_HOST debootstrap_args E user_name
     if [ ! -e "${DEPLOY_ISAR_BOOTSTRAP}" ]; then
         sudo rm -rf --one-file-system "${ROOTFSDIR}"
         if [ "${IS_HOST}" ];then
@@ -266,6 +268,7 @@ isar_bootstrap() {
 
         sudo -E -s <<'EOSUDO'
             set -e
+            BOOTSTRAP_ARCH=${HOST_ARCH}
             if [ ${IS_HOST} ]; then
                 ${DEBOOTSTRAP} $debootstrap_args \
                                ${@get_distro_components_argument(d, True)} \
@@ -281,6 +284,53 @@ isar_bootstrap() {
                                "${ROOTFSDIR}" \
                                "${@get_distro_source(d, False)}" \
                                ${DISTRO_DEBOOTSTRAP_SCRIPT}
+                BOOTSTRAP_ARCH=${DISTRO_ARCH}
+            fi
+
+            cat << EOF > "${SCHROOT_CONF}/chroot.d/${DEBDISTRONAME}-${user_name}-${DISTRO}-${BOOTSTRAP_ARCH}"
+[${DEBDISTRONAME}-${user_name}-${DISTRO}-${BOOTSTRAP_ARCH}]
+type=directory
+directory=${ROOTFSDIR}
+profile=${DEBDISTRONAME}-${user_name}
+users=${user_name}
+groups=root,sbuild
+root-users=${user_name}
+root-groups=root,sbuild
+source-root-users=${user_name}
+source-root-groups=root,sbuild
+union-type=overlay
+preserve-environment=true
+
+[${DEBDISTRONAME}-${user_name}-${DISTRO}-${BOOTSTRAP_ARCH}-rw]
+type=directory
+directory=${ROOTFSDIR}
+profile=${DEBDISTRONAME}-${user_name}
+users=${user_name}
+groups=root,sbuild
+root-users=${user_name}
+root-groups=root,sbuild
+preserve-environment=true
+EOF
+
+            mkdir -p '${ROOTFSDIR}/etc/apt/preferences.d'
+            cat << EOF > '${ROOTFSDIR}/etc/apt/preferences.d/isar-apt'
+Package: *
+Pin: release n=${DEBDISTRONAME}
+Pin-Priority: 1000
+EOF
+
+            # Prepare mount points
+            if [ ! -d "${SCHROOT_CONF}/${DEBDISTRONAME}-${user_name}" ]; then
+                cp -r "${SCHROOT_CONF}/sbuild" "${SCHROOT_CONF}/${DEBDISTRONAME}-${user_name}"
+            fi
+            sbuild_fstab="${SCHROOT_CONF}/${DEBDISTRONAME}-${user_name}/fstab"
+
+            fstab_isarapt="${DEPLOY_DIR}/isar-apt /isar-apt none rw,bind 0 0"
+            grep -qxF "${fstab_isarapt}" ${sbuild_fstab} || echo "${fstab_isarapt}" >> ${sbuild_fstab}
+
+            if [ -d ${DL_DIR} ]; then
+                fstab_downloads="${DL_DIR} /downloads none rw,bind 0 0"
+                grep -qxF "${fstab_downloads}" ${sbuild_fstab} || echo "${fstab_downloads}" >> ${sbuild_fstab}
             fi
 
             # Install apt config
-- 
2.25.1

[RFC 04/15] linux-module: Do not use shell environment

[Anton Mikanovich]

To make package build process independent of the shell environment we
should remove KDIR and PN passing through export call.
KDIR can be prepared during package build. This also will allow not to
rely on previous builddeps install task.
To pass PN variable we can just migrate to template-based debian/rules
file.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 .../files/debian/{rules => rules.tmpl}        | 10 +++++++-
 meta/recipes-kernel/linux-module/module.inc   | 23 +++++++------------
 2 files changed, 17 insertions(+), 16 deletions(-)
 rename meta/recipes-kernel/linux-module/files/debian/{rules => rules.tmpl} (55%)

diff --git a/meta/recipes-kernel/linux-module/files/debian/rules b/meta/recipes-kernel/linux-module/files/debian/rules.tmpl
similarity index 55%
rename from meta/recipes-kernel/linux-module/files/debian/rules
rename to meta/recipes-kernel/linux-module/files/debian/rules.tmpl
index c476bf3..39abc52 100755
--- a/meta/recipes-kernel/linux-module/files/debian/rules
+++ b/meta/recipes-kernel/linux-module/files/debian/rules.tmpl
@@ -24,11 +24,19 @@ ifneq (,$(findstring 86,$(DEB_HOST_GNU_CPU)))
 export ARCH=x86
 endif
 
+# Custom kernels contain the build folder directly.
+KDIR := $(shell dpkg -L linux-headers-${KERNEL_NAME} | grep "/lib/modules/.*/build")
+ifeq ($(KDIR),)
+# Debian kernels install that folder indirectly via a dependency.
+KERNEL_DEP := $(shell dpkg-query -W -f '$${Depends}' linux-headers-${KERNEL_NAME} | sed 's/.*\(linux-headers-[[:graph:]]*\).*/\1/')
+KDIR := $(shell dpkg -L $(KERNEL_DEP) | grep "/lib/modules/.*/build")
+endif
+
 override_dh_auto_build:
 	$(MAKE) -C $(KDIR) M=$(PWD) modules
 
 override_dh_auto_install:
-	$(MAKE) -C $(KDIR) M=$(PWD) INSTALL_MOD_PATH=$(PWD)/debian/$(PN) modules_install
+	$(MAKE) -C $(KDIR) M=$(PWD) INSTALL_MOD_PATH=$(PWD)/debian/${PN} modules_install
 
 %:
 	CFLAGS= LDFLAGS= dh $@ --parallel
diff --git a/meta/recipes-kernel/linux-module/module.inc b/meta/recipes-kernel/linux-module/module.inc
index 0515621..5b08623 100644
--- a/meta/recipes-kernel/linux-module/module.inc
+++ b/meta/recipes-kernel/linux-module/module.inc
@@ -22,8 +22,9 @@ AUTOLOAD ?= ""
 inherit dpkg
 
 TEMPLATE_FILES = "debian/control.tmpl \
-                  debian/changelog.tmpl"
-TEMPLATE_VARS += "KERNEL_NAME"
+                  debian/changelog.tmpl \
+                  debian/rules.tmpl"
+TEMPLATE_VARS += "KERNEL_NAME PN"
 
 do_prepare_build() {
     cp -r ${WORKDIR}/debian ${S}/
@@ -31,18 +32,10 @@ do_prepare_build() {
     for module in "${AUTOLOAD}"; do
         echo "echo $module >> /etc/modules" >> ${S}/debian/postinst
     done
-}
 
-dpkg_runbuild_prepend() {
-    # Custom kernels contain the build folder directly.
-    export KDIR=$(dpkg -L --root=${BUILDCHROOT_DIR} linux-headers-${KERNEL_NAME} | \
-                  grep "/lib/modules/.*/build")
-    if [ -z "$KDIR" ]; then
-        # Debian kernels install that folder indirectly via a dependency.
-        KERNEL_DEP=$(dpkg-query -W -f '${Depends}' --admindir=${BUILDCHROOT_DIR}/var/lib/dpkg \
-                     linux-headers-${KERNEL_NAME} | sed 's/.*\(linux-headers-[[:graph:]]*\).*/\1/')
-        export KDIR=$(dpkg -L --root=${BUILDCHROOT_DIR} ${KERNEL_DEP} | \
-                      grep "/lib/modules/.*/build")
-    fi
-    export PN=${PN}
+    # remove templates from the source tree
+    find ${S}/debian -name *.tmpl | xargs rm -f
+
+    # restore execute permissions
+    chmod a+x ${S}/debian/rules
 }
-- 
2.25.1

[RFC 05/15] u-boot: Do not use shell environment

[Anton Mikanovich]

To make package build process independent of the shell environment we
should remove U_BOOT_CONFIG and U_BOOT_BIN passing through export call.
So we migrate to template-based debian/rules file.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 .../u-boot/files/debian/{rules => rules.tmpl}          |  2 +-
 meta/recipes-bsp/u-boot/u-boot-custom.inc              | 10 ++++------
 2 files changed, 5 insertions(+), 7 deletions(-)
 rename meta/recipes-bsp/u-boot/files/debian/{rules => rules.tmpl} (93%)

diff --git a/meta/recipes-bsp/u-boot/files/debian/rules b/meta/recipes-bsp/u-boot/files/debian/rules.tmpl
similarity index 93%
rename from meta/recipes-bsp/u-boot/files/debian/rules
rename to meta/recipes-bsp/u-boot/files/debian/rules.tmpl
index 85bda91..0507fc5 100755
--- a/meta/recipes-bsp/u-boot/files/debian/rules
+++ b/meta/recipes-bsp/u-boot/files/debian/rules.tmpl
@@ -13,7 +13,7 @@ SET_CROSS_BUILD_TOOLS=CROSS_BUILD_TOOLS=y
 endif
 
 override_dh_auto_build:
-	$(MAKE) $(PARALLEL_MAKE) $(U_BOOT_CONFIG)
+	$(MAKE) $(PARALLEL_MAKE) ${U_BOOT_CONFIG}
 	$(MAKE) $(PARALLEL_MAKE) ${U_BOOT_BIN}
 	./scripts/get_default_envs.sh >u-boot-initial-env
 	$(MAKE) $(PARALLEL_MAKE) $(SET_CROSS_BUILD_TOOLS) NO_SDL=1 tools-only envtools
diff --git a/meta/recipes-bsp/u-boot/u-boot-custom.inc b/meta/recipes-bsp/u-boot/u-boot-custom.inc
index 9984d8c..819206a 100644
--- a/meta/recipes-bsp/u-boot/u-boot-custom.inc
+++ b/meta/recipes-bsp/u-boot/u-boot-custom.inc
@@ -26,8 +26,8 @@ python() {
 
 DEBIAN_BUILD_DEPENDS ?= "bc, bison, flex, device-tree-compiler, git"
 
-TEMPLATE_FILES = "debian/control.tmpl"
-TEMPLATE_VARS += "MACHINE DEBIAN_BUILD_DEPENDS"
+TEMPLATE_FILES = "debian/control.tmpl debian/rules.tmpl"
+TEMPLATE_VARS += "MACHINE DEBIAN_BUILD_DEPENDS U_BOOT_CONFIG U_BOOT_BIN"
 
 U_BOOT_TOOLS_PACKAGE ?= "0"
 U_BOOT_CONFIG_PACKAGE ?= "0"
@@ -70,9 +70,7 @@ u-boot-initial-env /etc
 fw_env.config      /etc
 EOF
     fi
-}
 
-dpkg_runbuild_prepend() {
-    export U_BOOT_CONFIG="${U_BOOT_CONFIG}"
-    export U_BOOT_BIN="${U_BOOT_BIN}"
+    # restore execute permissions
+    chmod a+x ${S}/debian/rules
 }
-- 
2.25.1

[RFC 06/15] trusted-firmware: Do not use shell environment

[Anton Mikanovich]

To make package build process independent of the shell environment we
should remove TF_A_PLATFORM and TF_A_EXTRA_BUILDARGS passing through
export call. So we migrate to template-based debian/rules file.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 .../files/debian/{rules => rules.tmpl}                 |  4 ++--
 .../trusted-firmware-a/trusted-firmware-a-custom.inc   | 10 ++++------
 2 files changed, 6 insertions(+), 8 deletions(-)
 rename meta/recipes-bsp/trusted-firmware-a/files/debian/{rules => rules.tmpl} (77%)

diff --git a/meta/recipes-bsp/trusted-firmware-a/files/debian/rules b/meta/recipes-bsp/trusted-firmware-a/files/debian/rules.tmpl
similarity index 77%
rename from meta/recipes-bsp/trusted-firmware-a/files/debian/rules
rename to meta/recipes-bsp/trusted-firmware-a/files/debian/rules.tmpl
index 70e1dd6..6dbf44d 100755
--- a/meta/recipes-bsp/trusted-firmware-a/files/debian/rules
+++ b/meta/recipes-bsp/trusted-firmware-a/files/debian/rules.tmpl
@@ -12,8 +12,8 @@ export CROSS_COMPILE=$(DEB_HOST_GNU_TYPE)-
 endif
 
 override_dh_auto_build:
-	CFLAGS= LDFLAGS= $(MAKE) $(PARALLEL_MAKE) PLAT=$(TF_A_PLATFORM) \
-		$(TF_A_EXTRA_BUILDARGS)
+	CFLAGS= LDFLAGS= $(MAKE) $(PARALLEL_MAKE) PLAT=${TF_A_PLATFORM} \
+		${TF_A_EXTRA_BUILDARGS}
 
 %:
 	dh $@
diff --git a/meta/recipes-bsp/trusted-firmware-a/trusted-firmware-a-custom.inc b/meta/recipes-bsp/trusted-firmware-a/trusted-firmware-a-custom.inc
index 1142bfc..3fe4c41 100644
--- a/meta/recipes-bsp/trusted-firmware-a/trusted-firmware-a-custom.inc
+++ b/meta/recipes-bsp/trusted-firmware-a/trusted-firmware-a-custom.inc
@@ -22,8 +22,8 @@ DEBIAN_BUILD_DEPENDS ?= ""
 
 PROVIDES += "trusted-firmware-a-${TF_A_NAME}"
 
-TEMPLATE_FILES = "debian/control.tmpl"
-TEMPLATE_VARS += "TF_A_NAME DEBIAN_BUILD_DEPENDS"
+TEMPLATE_FILES = "debian/control.tmpl debian/rules.tmpl"
+TEMPLATE_VARS += "TF_A_NAME DEBIAN_BUILD_DEPENDS TF_A_PLATFORM TF_A_EXTRA_BUILDARGS"
 
 do_prepare_build() {
     cp -r ${WORKDIR}/debian ${S}/
@@ -35,9 +35,7 @@ do_prepare_build() {
         echo "build/${TF_A_PLATFORM}/$binary /usr/lib/trusted-firmware-a/${TF_A_NAME}/" >> \
             ${S}/debian/trusted-firmware-a-${TF_A_NAME}.install
     done
-}
 
-dpkg_runbuild_prepend() {
-    export TF_A_PLATFORM="${TF_A_PLATFORM}"
-    export TF_A_EXTRA_BUILDARGS="${TF_A_EXTRA_BUILDARGS}"
+    # restore execute permissions
+    chmod a+x ${S}/debian/rules
 }
-- 
2.25.1

[RFC 07/15] optee-os: Do not use shell environment

[Anton Mikanovich]

To make package build process independent of the shell environment we
should remove OPTEE_PLATFORM and OPTEE_EXTRA_BUILDARGS passing through
export call. So we migrate to template-based debian/rules file.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 .../optee-os/files/debian/{rules => rules.tmpl}        |  4 ++--
 meta/recipes-bsp/optee-os/optee-os-custom.inc          | 10 ++++------
 2 files changed, 6 insertions(+), 8 deletions(-)
 rename meta/recipes-bsp/optee-os/files/debian/{rules => rules.tmpl} (75%)

diff --git a/meta/recipes-bsp/optee-os/files/debian/rules b/meta/recipes-bsp/optee-os/files/debian/rules.tmpl
similarity index 75%
rename from meta/recipes-bsp/optee-os/files/debian/rules
rename to meta/recipes-bsp/optee-os/files/debian/rules.tmpl
index d2e9900..9ab80df 100755
--- a/meta/recipes-bsp/optee-os/files/debian/rules
+++ b/meta/recipes-bsp/optee-os/files/debian/rules.tmpl
@@ -12,8 +12,8 @@ export CROSS_COMPILE=$(DEB_HOST_GNU_TYPE)-
 endif
 
 override_dh_auto_build:
-	CFLAGS= LDFLAGS= $(MAKE) $(PARALLEL_MAKE) PLATFORM=$(OPTEE_PLATFORM) \
-		$(OPTEE_EXTRA_BUILDARGS)
+	CFLAGS= LDFLAGS= $(MAKE) $(PARALLEL_MAKE) PLATFORM=${OPTEE_PLATFORM} \
+		${OPTEE_EXTRA_BUILDARGS}
 
 %:
 	dh $@
diff --git a/meta/recipes-bsp/optee-os/optee-os-custom.inc b/meta/recipes-bsp/optee-os/optee-os-custom.inc
index 1bd5196..e9eed54 100644
--- a/meta/recipes-bsp/optee-os/optee-os-custom.inc
+++ b/meta/recipes-bsp/optee-os/optee-os-custom.inc
@@ -22,8 +22,8 @@ DEBIAN_BUILD_DEPENDS ?= "python3-pycryptodome:native, python3-pyelftools"
 
 PROVIDES += "optee-os-${OPTEE_NAME}"
 
-TEMPLATE_FILES = "debian/control.tmpl"
-TEMPLATE_VARS += "OPTEE_NAME DEBIAN_BUILD_DEPENDS"
+TEMPLATE_FILES = "debian/control.tmpl debian/rules.tmpl"
+TEMPLATE_VARS += "OPTEE_NAME DEBIAN_BUILD_DEPENDS OPTEE_PLATFORM OPTEE_EXTRA_BUILDARGS"
 
 # split strip platform flavor, if any, from the specified platform string
 OPTEE_PLATFORM_BASE = "${@d.getVar('OPTEE_PLATFORM').split('-')[0]}"
@@ -38,9 +38,7 @@ do_prepare_build() {
         echo "out/arm-plat-${OPTEE_PLATFORM_BASE}/core/$binary /usr/lib/optee-os/${OPTEE_NAME}/" >> \
             ${S}/debian/optee-os-${OPTEE_NAME}.install
     done
-}
 
-dpkg_runbuild_prepend() {
-    export OPTEE_PLATFORM="${OPTEE_PLATFORM}"
-    export OPTEE_EXTRA_BUILDARGS="${OPTEE_EXTRA_BUILDARGS}"
+    # restore execute permissions
+    chmod a+x ${S}/debian/rules
 }
-- 
2.25.1

[RFC 08/15] kselftest: Do not use shell environment

[Anton Mikanovich]

To make package build process independent of the shell environment we
should remove KSELFTEST_ARGS passing through export call. The same
logic can be done by internal recipe parser and then pass variable to
template-based debian/rules file.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 .../kselftest/files/{rules => rules.tmpl}     |  0
 meta/recipes-kernel/kselftest/kselftest.inc   | 21 +++++++------------
 2 files changed, 8 insertions(+), 13 deletions(-)
 rename meta/recipes-kernel/kselftest/files/{rules => rules.tmpl} (100%)

diff --git a/meta/recipes-kernel/kselftest/files/rules b/meta/recipes-kernel/kselftest/files/rules.tmpl
similarity index 100%
rename from meta/recipes-kernel/kselftest/files/rules
rename to meta/recipes-kernel/kselftest/files/rules.tmpl
diff --git a/meta/recipes-kernel/kselftest/kselftest.inc b/meta/recipes-kernel/kselftest/kselftest.inc
index 2a76028..6187d8e 100644
--- a/meta/recipes-kernel/kselftest/kselftest.inc
+++ b/meta/recipes-kernel/kselftest/kselftest.inc
@@ -33,26 +33,21 @@ DEBIAN_BUILD_DEPENDS ?= " \
     llvm:native, \
 "
 
-SRC_URI += "file://rules"
+SRC_URI += "file://rules.tmpl"
 S = "${WORKDIR}/linux-${PV}"
 
+TEMPLATE_FILES = "rules.tmpl"
+TEMPLATE_VARS += "KSELFTEST_ARGS"
+
 KSELFTEST_TARGETS ?= ""
 KSELFTEST_SKIP_TARGETS ?= ""
 KSELFTEST_FORCE_TARGETS ?= "0"
 
+KSELFTEST_ARGS = "${@ "TARGETS=\"${KSELFTEST_TARGETS}\"" if d.getVar('KSELFTEST_TARGETS', True) else ''}"
+KSELFTEST_ARGS_append = "${@ " FORCE_TARGETS=1" if d.getVar('KSELFTEST_FORCE_TARGETS', True) == '1' else ''}"
+KSELFTEST_ARGS_append .= "${@ " SKIP_TARGETS=\"${KSELFTEST_SKIP_TARGETS}\"" if d.getVar('KSELFTEST_SKIP_TARGETS', True) else ''}"
+
 do_prepare_build[cleandirs] += "${S}/debian"
 do_prepare_build() {
     deb_debianize
 }
-
-dpkg_runbuild_prepend() {
-    if [ -n "${KSELFTEST_TARGETS}" ];then
-        export KSELFTEST_ARGS="TARGETS=\"${KSELFTEST_TARGETS}\""
-    fi
-    if [ ${KSELFTEST_FORCE_TARGETS} -eq 1 ];then
-        export KSELFTEST_ARGS="${KSELFTEST_ARGS} FORCE_TARGETS=1"
-    fi
-    if [ -n "${KSELFTEST_SKIP_TARGETS}" ];then
-        export KSELFTEST_ARGS="${KSELFTEST_ARGS} SKIP_TARGETS=\"${KSELFTEST_SKIP_TARGETS}\""
-    fi
-}
-- 
2.25.1

[RFC 09/15] dpkg: Build packages with sbuild

[Anton Mikanovich]

Use previously exported to schroot rootfs to build the package with
sbuild tool. Provide isar-apt as extra repository to be automatically
add to apt sources.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/dpkg.bbclass | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 29e2b89..524aecc 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -2,6 +2,7 @@
 # Copyright (C) 2015-2018 ilbers GmbH
 
 inherit dpkg-base
+inherit sbuild
 
 PACKAGE_ARCH ?= "${DISTRO_ARCH}"
 
@@ -34,10 +35,16 @@ do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
 
 addtask devshell after do_install_builddeps
 
+ISAR_APT_REPO ?= "deb [trusted=yes] file:///isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO} ${DEBDISTRONAME} main"
+
 # Build package from sources using build script
 dpkg_runbuild() {
     E="${@ isar_export_proxies(d)}"
     export PARALLEL_MAKE="${PARALLEL_MAKE}"
-    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
-         /isar/build.sh ${PP}/${PPS} ${PACKAGE_ARCH}
+
+    sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
+        --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \
+        --starting-build-commands="runuser -u ${@os.getlogin()} -- sh -c \"${SBUILD_PREBUILD:-:}\"" \
+        --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
+        --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
 }
-- 
2.25.1

[RFC 10/15] sbuild: Introduce environment variables export API

[Anton Mikanovich]

In case there is no ability to pass variables through template
mechanism we need to keep fallback API to migrate from bare export.
Using it with optee-os-stm32mp15x recipe as example.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 .../optee-os/optee-os-stm32mp15x_3.11.0.bb          |  2 +-
 meta/classes/sbuild.bbclass                         | 13 +++++++++++++
 2 files changed, 14 insertions(+), 1 deletion(-)

diff --git a/meta-isar/recipes-bsp/optee-os/optee-os-stm32mp15x_3.11.0.bb b/meta-isar/recipes-bsp/optee-os/optee-os-stm32mp15x_3.11.0.bb
index 0fa2005..19aa763 100644
--- a/meta-isar/recipes-bsp/optee-os/optee-os-stm32mp15x_3.11.0.bb
+++ b/meta-isar/recipes-bsp/optee-os/optee-os-stm32mp15x_3.11.0.bb
@@ -21,5 +21,5 @@ OPTEE_BINARIES = "tee-header_v2.stm32 tee-pageable_v2.stm32 tee-pager_v2.stm32"
 # Set version manually to PV, the tarball does not contain any hint.
 # Alternative: pull from git and add git as build dependency.
 dpkg_runbuild_prepend() {
-    export TEE_IMPL_VERSION=${PV}
+    sbuild_export TEE_IMPL_VERSION=${PV}
 }
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index ed2d0cf..cc7db66 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -15,3 +15,16 @@ python __anonymous() {
 
 SBUILD_CHROOT ?= "${DEBDISTRONAME}-${@os.getlogin()}-${DISTRO}-${SBUILD_HOST_ARCH}"
 SBUILD_CHROOT_RW ?= "${DEBDISTRONAME}-${@os.getlogin()}-${DISTRO}-${SBUILD_HOST_ARCH}-rw"
+
+sbuild_export() {
+    SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
+    VAR_LINE="'${1%%=*}' => '${1#*=}',"
+    if [ -s "${SBUILD_CONFIG}" ]; then
+        sed -i -e "\$i\\" -e "${VAR_LINE}" ${SBUILD_CONFIG}
+    else
+        echo "\$build_environment = {" > ${SBUILD_CONFIG}
+        echo "${VAR_LINE}" >> ${SBUILD_CONFIG}
+        echo "};" >> ${SBUILD_CONFIG}
+    fi
+    export SBUILD_CONFIG="${SBUILD_CONFIG}"
+}
-- 
2.25.1

[RFC 11/15] dpkg: Remove builddeps install task

[Anton Mikanovich]

Build dependencies are installed by sbuild now.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/dpkg-gbp.bbclass |  3 ++-
 meta/classes/dpkg.bbclass     | 29 +----------------------------
 2 files changed, 3 insertions(+), 29 deletions(-)

diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index 2711e6c..0c60bda 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -12,7 +12,7 @@ PATCHTOOL ?= "git"
 GBP_DEPENDS ?= "git-buildpackage pristine-tar"
 GBP_EXTRA_OPTIONS ?= "--pristine-tar"
 
-builddeps_install_append() {
+do_prepare_build() {
     deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
     sudo -E chroot ${BUILDCHROOT_DIR} \
         apt-get install -y -o Debug::pkgProblemResolver=yes \
@@ -22,6 +22,7 @@ builddeps_install_append() {
         apt-get install -y -o Debug::pkgProblemResolver=yes \
                         --no-install-recommends ${GBP_DEPENDS}
 }
+do_prepare_build[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
 
 dpkg_runbuild_prepend() {
     sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 524aecc..582c2b6 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -6,34 +6,7 @@ inherit sbuild
 
 PACKAGE_ARCH ?= "${DISTRO_ARCH}"
 
-# Install build dependencies for package
-builddeps_install() {
-    distro="${DISTRO}"
-    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
-       distro="${HOST_DISTRO}"
-    fi
-    deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
-        ${PP}/${PPS} ${PACKAGE_ARCH} --download-only
-    deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
-        ${PP}/${PPS} ${PACKAGE_ARCH}
-}
-
-python do_install_builddeps() {
-    dpkg_do_mounts(d)
-    isar_export_proxies(d)
-    try:
-        bb.build.exec_func("builddeps_install", d)
-    finally:
-        dpkg_undo_mounts(d)
-}
-
-addtask install_builddeps after do_prepare_build before do_dpkg_build
-# apt and reprepro may not run in parallel, acquire the Isar lock
-do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
-
-addtask devshell after do_install_builddeps
+addtask devshell after do_prepare_build
 
 ISAR_APT_REPO ?= "deb [trusted=yes] file:///isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO} ${DEBDISTRONAME} main"
 
-- 
2.25.1

[RFC 12/15] dpkg-gbp: Migrate to schroot

[Anton Mikanovich]

Use schroot instead of chroot for git-buildpackage.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/dpkg-gbp.bbclass | 15 ++++-------
 meta/classes/sbuild.bbclass   | 48 +++++++++++++++++++++++++++++++++++
 2 files changed, 53 insertions(+), 10 deletions(-)

diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index 0c60bda..e4e5ea6 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -13,18 +13,13 @@ GBP_DEPENDS ?= "git-buildpackage pristine-tar"
 GBP_EXTRA_OPTIONS ?= "--pristine-tar"
 
 do_prepare_build() {
-    deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} \
-        apt-get install -y -o Debug::pkgProblemResolver=yes \
-                        --no-install-recommends --download-only ${GBP_DEPENDS}
-    deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} \
-        apt-get install -y -o Debug::pkgProblemResolver=yes \
-                        --no-install-recommends ${GBP_DEPENDS}
+    schroot_install "${GBP_DEPENDS}"
 }
 do_prepare_build[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
 
+SCHROOT_MOUNTS = "${WORKDIR}:${PP}"
+
 dpkg_runbuild_prepend() {
-    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
-        sh -c "cd ${PP}/${PPS} && gbp export-orig ${GBP_EXTRA_OPTIONS}"
+    schroot_run -d ${PP}/${PPS} -c ${SBUILD_CHROOT} -- \
+        gbp export-orig ${GBP_EXTRA_OPTIONS}
 }
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index cc7db66..3eede71 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -3,6 +3,8 @@
 
 SCHROOT_CONF ?= "/etc/schroot"
 
+SCHROOT_MOUNTS ?= ""
+
 python __anonymous() {
     mode = d.getVar('ISAR_CROSS_COMPILE', True)
     distro_arch = d.getVar('DISTRO_ARCH')
@@ -28,3 +30,49 @@ sbuild_export() {
     fi
     export SBUILD_CONFIG="${SBUILD_CONFIG}"
 }
+
+schroot_install() {
+    APTS="$1"
+    #TODO deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
+    schroot -d / -c ${SBUILD_CHROOT_RW} -u root -- \
+        apt install -y -o Debug::pkgProblemResolver=yes \
+                    --no-install-recommends --download-only ${APTS}
+    #TODO deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
+    schroot -d / -c ${SBUILD_CHROOT_RW} -u root -- \
+        apt install -y -o Debug::pkgProblemResolver=yes \
+                    --no-install-recommends ${APTS}
+}
+
+SCHROOT_FSTAB ?= "${SCHROOT_CONF}/${DEBDISTRONAME}-${@os.getlogin()}/fstab"
+SCHROOT_LOCKFILE ?= "schroot_fstab.lock"
+
+insert_mounts() {
+    sudo -s <<'EOSUDO'
+        ( flock 9
+        set -e
+        for mp in ${SCHROOT_MOUNTS}; do
+            FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
+            grep -qxF "${FSTAB_LINE}" ${SCHROOT_FSTAB} || \
+                echo "${FSTAB_LINE}" >> ${SCHROOT_FSTAB}
+        done
+        ) 9>'${SCHROOT_LOCKFILE}'
+EOSUDO
+}
+
+remove_mounts() {
+    sudo -s <<'EOSUDO'
+        ( flock 9
+        set -e
+        for mp in ${SCHROOT_MOUNTS}; do
+            FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
+            sed -i "\|${FSTAB_LINE}|d" ${SCHROOT_FSTAB}
+        done
+        ) 9>'${SCHROOT_LOCKFILE}'
+EOSUDO
+}
+
+schroot_run() {
+    insert_mounts
+    schroot $@
+    remove_mounts
+}
-- 
2.25.1

[RFC 13/15] img: Use schroot for image prepare

[Anton Mikanovich]

Perform all image generation functions inside schroot.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/ext4-img.bbclass              |  8 ++------
 meta/classes/fit-img.bbclass               | 11 ++++-------
 meta/classes/image-tools-extension.bbclass | 14 ++++----------
 meta/classes/image.bbclass                 | 19 ++-----------------
 meta/classes/ubi-img.bbclass               | 11 ++++-------
 meta/classes/ubifs-img.bbclass             | 11 ++++-------
 6 files changed, 20 insertions(+), 54 deletions(-)

diff --git a/meta/classes/ext4-img.bbclass b/meta/classes/ext4-img.bbclass
index 26e0137..6f89fce 100644
--- a/meta/classes/ext4-img.bbclass
+++ b/meta/classes/ext4-img.bbclass
@@ -13,12 +13,8 @@ do_ext4_image() {
 
     truncate -s ${ROOTFS_SIZE}K '${DEPLOY_DIR_IMAGE}/${EXT4_IMAGE_FILE}'
 
-    image_do_mounts
-
-    sudo chroot ${BUILDCHROOT_DIR} /sbin/mke2fs ${MKE2FS_ARGS} \
-                -F -d '${PP_ROOTFS}' '${PP_DEPLOY}/${EXT4_IMAGE_FILE}'
-
-    image_undo_mounts
+    schroot_run -d ${PP_WORK} -c ${SBUILD_CHROOT} -u root -- /sbin/mke2fs \
+        ${MKE2FS_ARGS} -F -d '${PP_ROOTFS}' '${PP_DEPLOY}/${EXT4_IMAGE_FILE}'
 }
 
 addtask ext4_image before do_image after do_image_tools
diff --git a/meta/classes/fit-img.bbclass b/meta/classes/fit-img.bbclass
index da8e1da..483ef66 100644
--- a/meta/classes/fit-img.bbclass
+++ b/meta/classes/fit-img.bbclass
@@ -18,12 +18,9 @@ do_fit_image() {
 
     rm -f '${DEPLOY_DIR_IMAGE}/${FIT_IMAGE_FILE}'
 
-    image_do_mounts
-
-    # Create fit image using buildchroot tools
-    sudo chroot ${BUILDCHROOT_DIR} /usr/bin/mkimage ${MKIMAGE_ARGS} \
-                -f '${PP_WORK}/${FIT_IMAGE_SOURCE}' '${PP_DEPLOY}/${FIT_IMAGE_FILE}'
-
-    image_undo_mounts
+    # Create fit image using schroot tools
+    schroot_run -d ${PP_WORK} -c ${SBUILD_CHROOT} -- \
+        /usr/bin/mkimage ${MKIMAGE_ARGS} -f '${PP_WORK}/${FIT_IMAGE_SOURCE}' \
+        '${PP_DEPLOY}/${FIT_IMAGE_FILE}'
 }
 addtask fit_image before do_image after do_image_tools do_transform_template
diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass
index 4738479..7896695 100644
--- a/meta/classes/image-tools-extension.bbclass
+++ b/meta/classes/image-tools-extension.bbclass
@@ -8,8 +8,6 @@
 # Imager are expected to run natively, thus will use the target buildchroot.
 ISAR_CROSS_COMPILE = "0"
 
-inherit buildchroot
-
 IMAGER_INSTALL ??= ""
 IMAGER_BUILD_DEPS ??= ""
 DEPENDS += "${IMAGER_BUILD_DEPS}"
@@ -22,11 +20,9 @@ do_install_imager_deps() {
         exit
     fi
 
-    buildchroot_do_mounts
-
     E="${@ isar_export_proxies(d)}"
-    deb_dl_dir_import ${BUILDCHROOT_DIR} ${DISTRO}
-    sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
+    #TODO deb_dl_dir_import ${BUILDCHROOT_DIR} ${DISTRO}
+    schroot -d / -c ${SBUILD_CHROOT_RW} -u root -- sh -c ' \
         apt-get update \
             -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
             -o Dir::Etc::SourceParts="-" \
@@ -35,12 +31,10 @@ do_install_imager_deps() {
             --allow-unauthenticated --allow-downgrades --download-only install \
             ${IMAGER_INSTALL}'
 
-    deb_dl_dir_export ${BUILDCHROOT_DIR} ${DISTRO}
-    sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
+    #TODO deb_dl_dir_export ${BUILDCHROOT_DIR} ${DISTRO}
+    schroot -d / -c ${SBUILD_CHROOT_RW} -u root -- sh -c ' \
         apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y \
             --allow-unauthenticated --allow-downgrades install \
             ${IMAGER_INSTALL}'
-
-    buildchroot_undo_mounts
 }
 addtask install_imager_deps before do_image_tools
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 12d1616..9ca484a 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -52,24 +52,9 @@ DEPENDS += "${IMAGE_INSTALL}"
 ISAR_RELEASE_CMD_DEFAULT = "git -C ${LAYERDIR_core} describe --tags --dirty --match 'v[0-9].[0-9]*'"
 ISAR_RELEASE_CMD ?= "${ISAR_RELEASE_CMD_DEFAULT}"
 
-image_do_mounts() {
-    sudo flock ${MOUNT_LOCKFILE} -c ' \
-        mkdir -p "${BUILDROOT_DEPLOY}" "${BUILDROOT_ROOTFS}" "${BUILDROOT_WORK}"
-        mount --bind "${DEPLOY_DIR_IMAGE}" "${BUILDROOT_DEPLOY}"
-        mount --bind "${IMAGE_ROOTFS}" "${BUILDROOT_ROOTFS}"
-        mount --bind "${WORKDIR}" "${BUILDROOT_WORK}"
-    '
-    buildchroot_do_mounts
-}
+inherit sbuild
 
-image_undo_mounts() {
-    buildchroot_undo_mounts
-    sudo flock ${MOUNT_LOCKFILE} -c ' \
-        umount "${BUILDROOT_DEPLOY}"
-        umount "${BUILDROOT_ROOTFS}"
-        umount "${BUILDROOT_WORK}"
-    '
-}
+SCHROOT_MOUNTS = "${DEPLOY_DIR_IMAGE}:${PP_DEPLOY} ${IMAGE_ROOTFS}:${PP_ROOTFS} ${WORKDIR}:${PP_WORK}"
 
 ROOTFSDIR = "${IMAGE_ROOTFS}"
 ROOTFS_FEATURES += "clean-package-cache generate-manifest export-dpkg-status"
diff --git a/meta/classes/ubi-img.bbclass b/meta/classes/ubi-img.bbclass
index 0f4c4cb..a753504 100644
--- a/meta/classes/ubi-img.bbclass
+++ b/meta/classes/ubi-img.bbclass
@@ -21,12 +21,9 @@ do_ubi_image() {
 
     rm -f '${DEPLOY_DIR_IMAGE}/${UBI_IMAGE_FILE}'
 
-    image_do_mounts
-
-    # Create ubi image using buildchroot tools
-    sudo chroot ${BUILDCHROOT_DIR} /usr/sbin/ubinize ${UBINIZE_ARGS} \
-                -o '${PP_DEPLOY}/${UBI_IMAGE_FILE}' '${PP_WORK}/${UBINIZE_CFG}'
-
-    image_undo_mounts
+    # Create ubi image using schroot tools
+    schroot_run -d ${PP_WORK} -c ${SBUILD_CHROOT} -- \
+        /usr/sbin/ubinize ${UBINIZE_ARGS} -o '${PP_DEPLOY}/${UBI_IMAGE_FILE}' \
+        '${PP_WORK}/${UBINIZE_CFG}'
 }
 addtask ubi_image before do_image after do_image_tools do_transform_template
diff --git a/meta/classes/ubifs-img.bbclass b/meta/classes/ubifs-img.bbclass
index e422b46..13ce667 100644
--- a/meta/classes/ubifs-img.bbclass
+++ b/meta/classes/ubifs-img.bbclass
@@ -16,13 +16,10 @@ IMAGER_INSTALL += "mtd-utils"
 do_ubifs_image() {
     rm -f '${DEPLOY_DIR_IMAGE}/${UBIFS_IMAGE_FILE}'
 
-    image_do_mounts
-
-    # Create ubifs image using buildchroot tools
-    sudo chroot ${BUILDCHROOT_DIR} /usr/sbin/mkfs.ubifs ${MKUBIFS_ARGS} \
-                -r '${PP_ROOTFS}' '${PP_DEPLOY}/${UBIFS_IMAGE_FILE}'
-
-    image_undo_mounts
+    # Create ubifs image using schroot tools
+    schroot_run -d ${PP_WORK} -c ${SBUILD_CHROOT} -- \
+        /usr/sbin/mkfs.ubifs ${MKUBIFS_ARGS} -r '${PP_ROOTFS}' \
+        '${PP_DEPLOY}/${UBIFS_IMAGE_FILE}'
 }
 
 addtask ubifs_image before do_image after do_image_tools
-- 
2.25.1

[RFC 14/15] linux-mainline: Move cfg fragment test to debian/rules

[Anton Mikanovich]

Perform all config fragments checking in Debian way.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 .../linux/linux-mainline_5.4.70.bb               | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)

diff --git a/meta-isar/recipes-kernel/linux/linux-mainline_5.4.70.bb b/meta-isar/recipes-kernel/linux/linux-mainline_5.4.70.bb
index 980e137..28e51c0 100644
--- a/meta-isar/recipes-kernel/linux/linux-mainline_5.4.70.bb
+++ b/meta-isar/recipes-kernel/linux/linux-mainline_5.4.70.bb
@@ -25,11 +25,13 @@ LINUX_VERSION_EXTENSION = "-isar"
 
 # For testing purposes only
 dpkg_configure_kernel_append() {
-    if ! grep "# CONFIG_MTD is not set" ${S}/${KERNEL_BUILD_DIR}/.config && \
-       ! grep "# CONFIG_MTD_UBI is not set" ${S}/${KERNEL_BUILD_DIR}/.config; then
-        grep "# CONFIG_UBIFS_FS is not set" ${S}/${KERNEL_BUILD_DIR}/.config || \
-            bbfatal "Self-check failed: CONFIG_UBIFS_FS still enabled"
-    fi
-    grep "CONFIG_ROOT_NFS=y" ${S}/${KERNEL_BUILD_DIR}/.config || \
-        bbfatal "Self-check failed: CONFIG_ROOT_NFS not enabled"
+cat << EOF | sed -i '/^override_dh_auto_build/ r /dev/stdin' ${S}/debian/rules
+	if ! grep "# CONFIG_MTD is not set" \$(O)/.config && \\
+	   ! grep "# CONFIG_MTD_UBI is not set" \$(O)/.config; then \\
+	    grep "# CONFIG_UBIFS_FS is not set" \$(O)/.config || \\
+	        (echo "Self-check failed: CONFIG_UBIFS_FS still enabled" && exit 1); \\
+	fi
+	grep "CONFIG_ROOT_NFS=y" \$(O)/.config || \\
+	    (echo "Self-check failed: CONFIG_ROOT_NFS not enabled" && exit 1)
+EOF
 }
-- 
2.25.1

[RFC 15/15] linux-custom: Prepare kernel config inside sbuild

[Anton Mikanovich]

Move prepare code inside sbuild hook to perform all the operations
inside schroot.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/recipes-kernel/linux/linux-custom.inc | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index ed89aa0..2111584 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -125,6 +125,9 @@ do_prepare_build_prepend() {
 	# remove templates from the source tree
 	find ${S}/debian -name *.tmpl | xargs rm -f
 
+	# restore execute permissions
+	chmod a+x ${S}/debian/rules
+
 	# rename install/remove hooks to match user-specified name for our linux-image package
 	mv ${S}/debian/linux-image.postinst ${S}/debian/linux-image-${KERNEL_NAME_PROVIDED}.postinst
 	mv ${S}/debian/linux-image.postrm ${S}/debian/linux-image-${KERNEL_NAME_PROVIDED}.postrm
@@ -164,9 +167,9 @@ dpkg_configure_kernel() {
 		(cd ${WORKDIR} && cp ${src_frags} ${S}/debian/fragments/)
 	fi
 
-	sudo -E chroot --userspec=$(id -u):$(id -g) ${BUILDCHROOT_DIR} sh -c " \
+	SBUILD_PREBUILD="                                                      \
 		export ARCH=${KERNEL_ARCH} &&                                  \
-		cd ${PP}/${PPS} &&                                             \
+		cd %p &&                                                       \
 		make O=${KERNEL_BUILD_DIR} ${config_target} &&                 \
 		./scripts/kconfig/merge_config.sh                              \
 			-O ${KERNEL_BUILD_DIR}/                                \
-- 
2.25.1

Re: [RFC 04/15] linux-module: Do not use shell environment

[Jan Kiszka]

On 26.07.21 15:46, Anton Mikanovich wrote:
> To make package build process independent of the shell environment we
> should remove KDIR and PN passing through export call.
> KDIR can be prepared during package build. This also will allow not to
> rely on previous builddeps install task.
> To pass PN variable we can just migrate to template-based debian/rules
> file.
> 
> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> ---
>  .../files/debian/{rules => rules.tmpl}        | 10 +++++++-
>  meta/recipes-kernel/linux-module/module.inc   | 23 +++++++------------
>  2 files changed, 17 insertions(+), 16 deletions(-)
>  rename meta/recipes-kernel/linux-module/files/debian/{rules => rules.tmpl} (55%)
> 
> diff --git a/meta/recipes-kernel/linux-module/files/debian/rules b/meta/recipes-kernel/linux-module/files/debian/rules.tmpl
> similarity index 55%
> rename from meta/recipes-kernel/linux-module/files/debian/rules
> rename to meta/recipes-kernel/linux-module/files/debian/rules.tmpl
> index c476bf3..39abc52 100755
> --- a/meta/recipes-kernel/linux-module/files/debian/rules
> +++ b/meta/recipes-kernel/linux-module/files/debian/rules.tmpl
> @@ -24,11 +24,19 @@ ifneq (,$(findstring 86,$(DEB_HOST_GNU_CPU)))
>  export ARCH=x86
>  endif
>  
> +# Custom kernels contain the build folder directly.
> +KDIR := $(shell dpkg -L linux-headers-${KERNEL_NAME} | grep "/lib/modules/.*/build")
> +ifeq ($(KDIR),)
> +# Debian kernels install that folder indirectly via a dependency.
> +KERNEL_DEP := $(shell dpkg-query -W -f '$${Depends}' linux-headers-${KERNEL_NAME} | sed 's/.*\(linux-headers-[[:graph:]]*\).*/\1/')
> +KDIR := $(shell dpkg -L $(KERNEL_DEP) | grep "/lib/modules/.*/build")
> +endif
> +
>  override_dh_auto_build:
>  	$(MAKE) -C $(KDIR) M=$(PWD) modules
>  
>  override_dh_auto_install:
> -	$(MAKE) -C $(KDIR) M=$(PWD) INSTALL_MOD_PATH=$(PWD)/debian/$(PN) modules_install
> +	$(MAKE) -C $(KDIR) M=$(PWD) INSTALL_MOD_PATH=$(PWD)/debian/${PN} modules_install
>  
>  %:
>  	CFLAGS= LDFLAGS= dh $@ --parallel
> diff --git a/meta/recipes-kernel/linux-module/module.inc b/meta/recipes-kernel/linux-module/module.inc
> index 0515621..5b08623 100644
> --- a/meta/recipes-kernel/linux-module/module.inc
> +++ b/meta/recipes-kernel/linux-module/module.inc
> @@ -22,8 +22,9 @@ AUTOLOAD ?= ""
>  inherit dpkg
>  
>  TEMPLATE_FILES = "debian/control.tmpl \
> -                  debian/changelog.tmpl"
> -TEMPLATE_VARS += "KERNEL_NAME"
> +                  debian/changelog.tmpl \
> +                  debian/rules.tmpl"
> +TEMPLATE_VARS += "KERNEL_NAME PN"
>  
>  do_prepare_build() {
>      cp -r ${WORKDIR}/debian ${S}/
> @@ -31,18 +32,10 @@ do_prepare_build() {
>      for module in "${AUTOLOAD}"; do
>          echo "echo $module >> /etc/modules" >> ${S}/debian/postinst
>      done
> -}
>  
> -dpkg_runbuild_prepend() {
> -    # Custom kernels contain the build folder directly.
> -    export KDIR=$(dpkg -L --root=${BUILDCHROOT_DIR} linux-headers-${KERNEL_NAME} | \
> -                  grep "/lib/modules/.*/build")
> -    if [ -z "$KDIR" ]; then
> -        # Debian kernels install that folder indirectly via a dependency.
> -        KERNEL_DEP=$(dpkg-query -W -f '${Depends}' --admindir=${BUILDCHROOT_DIR}/var/lib/dpkg \
> -                     linux-headers-${KERNEL_NAME} | sed 's/.*\(linux-headers-[[:graph:]]*\).*/\1/')
> -        export KDIR=$(dpkg -L --root=${BUILDCHROOT_DIR} ${KERNEL_DEP} | \
> -                      grep "/lib/modules/.*/build")
> -    fi
> -    export PN=${PN}
> +    # remove templates from the source tree
> +    find ${S}/debian -name *.tmpl | xargs rm -f
> +
> +    # restore execute permissions
> +    chmod a+x ${S}/debian/rules
>  }
> 

Exporting build vars via the shell is a fairly common pattern by now,
not only in the many recipes and includes you had to patch in isar.
Can't we do something that people do not have to fork rules files into
rules.tmpl in order to inject variables set in recipes?

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [RFC 10/15] sbuild: Introduce environment variables export API

[Jan Kiszka]

On 26.07.21 15:46, Anton Mikanovich wrote:
> In case there is no ability to pass variables through template
> mechanism we need to keep fallback API to migrate from bare export.
> Using it with optee-os-stm32mp15x recipe as example.
> 

Ah, here is what I was looking for.

> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> ---
>  .../optee-os/optee-os-stm32mp15x_3.11.0.bb          |  2 +-
>  meta/classes/sbuild.bbclass                         | 13 +++++++++++++
>  2 files changed, 14 insertions(+), 1 deletion(-)
> 
> diff --git a/meta-isar/recipes-bsp/optee-os/optee-os-stm32mp15x_3.11.0.bb b/meta-isar/recipes-bsp/optee-os/optee-os-stm32mp15x_3.11.0.bb
> index 0fa2005..19aa763 100644
> --- a/meta-isar/recipes-bsp/optee-os/optee-os-stm32mp15x_3.11.0.bb
> +++ b/meta-isar/recipes-bsp/optee-os/optee-os-stm32mp15x_3.11.0.bb
> @@ -21,5 +21,5 @@ OPTEE_BINARIES = "tee-header_v2.stm32 tee-pageable_v2.stm32 tee-pager_v2.stm32"
>  # Set version manually to PV, the tarball does not contain any hint.
>  # Alternative: pull from git and add git as build dependency.
>  dpkg_runbuild_prepend() {
> -    export TEE_IMPL_VERSION=${PV}
> +    sbuild_export TEE_IMPL_VERSION=${PV}

Maybe not call this mechanism "sbuild" - what if we switch the builder
in the future?

And it should be documented.

>  }
> diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
> index ed2d0cf..cc7db66 100644
> --- a/meta/classes/sbuild.bbclass
> +++ b/meta/classes/sbuild.bbclass
> @@ -15,3 +15,16 @@ python __anonymous() {
>  
>  SBUILD_CHROOT ?= "${DEBDISTRONAME}-${@os.getlogin()}-${DISTRO}-${SBUILD_HOST_ARCH}"
>  SBUILD_CHROOT_RW ?= "${DEBDISTRONAME}-${@os.getlogin()}-${DISTRO}-${SBUILD_HOST_ARCH}-rw"
> +
> +sbuild_export() {
> +    SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
> +    VAR_LINE="'${1%%=*}' => '${1#*=}',"
> +    if [ -s "${SBUILD_CONFIG}" ]; then
> +        sed -i -e "\$i\\" -e "${VAR_LINE}" ${SBUILD_CONFIG}
> +    else
> +        echo "\$build_environment = {" > ${SBUILD_CONFIG}
> +        echo "${VAR_LINE}" >> ${SBUILD_CONFIG}
> +        echo "};" >> ${SBUILD_CONFIG}
> +    fi
> +    export SBUILD_CONFIG="${SBUILD_CONFIG}"
> +}
> 

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [RFC 14/15] linux-mainline: Move cfg fragment test to debian/rules

[Jan Kiszka]

On 26.07.21 15:46, Anton Mikanovich wrote:
> Perform all config fragments checking in Debian way.
> 

As always: A commit message documents the "why", not so much the "what".

> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> ---
>  .../linux/linux-mainline_5.4.70.bb               | 16 +++++++++-------
>  1 file changed, 9 insertions(+), 7 deletions(-)
> 
> diff --git a/meta-isar/recipes-kernel/linux/linux-mainline_5.4.70.bb b/meta-isar/recipes-kernel/linux/linux-mainline_5.4.70.bb
> index 980e137..28e51c0 100644
> --- a/meta-isar/recipes-kernel/linux/linux-mainline_5.4.70.bb
> +++ b/meta-isar/recipes-kernel/linux/linux-mainline_5.4.70.bb
> @@ -25,11 +25,13 @@ LINUX_VERSION_EXTENSION = "-isar"
>  
>  # For testing purposes only
>  dpkg_configure_kernel_append() {
> -    if ! grep "# CONFIG_MTD is not set" ${S}/${KERNEL_BUILD_DIR}/.config && \
> -       ! grep "# CONFIG_MTD_UBI is not set" ${S}/${KERNEL_BUILD_DIR}/.config; then
> -        grep "# CONFIG_UBIFS_FS is not set" ${S}/${KERNEL_BUILD_DIR}/.config || \
> -            bbfatal "Self-check failed: CONFIG_UBIFS_FS still enabled"
> -    fi
> -    grep "CONFIG_ROOT_NFS=y" ${S}/${KERNEL_BUILD_DIR}/.config || \
> -        bbfatal "Self-check failed: CONFIG_ROOT_NFS not enabled"
> +cat << EOF | sed -i '/^override_dh_auto_build/ r /dev/stdin' ${S}/debian/rules
> +	if ! grep "# CONFIG_MTD is not set" \$(O)/.config && \\
> +	   ! grep "# CONFIG_MTD_UBI is not set" \$(O)/.config; then \\
> +	    grep "# CONFIG_UBIFS_FS is not set" \$(O)/.config || \\
> +	        (echo "Self-check failed: CONFIG_UBIFS_FS still enabled" && exit 1); \\
> +	fi
> +	grep "CONFIG_ROOT_NFS=y" \$(O)/.config || \\
> +	    (echo "Self-check failed: CONFIG_ROOT_NFS not enabled" && exit 1)
> +EOF
>  }
> 

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [RFC 05/15] u-boot: Do not use shell environment

[Jan Kiszka]

On 26.07.21 15:46, Anton Mikanovich wrote:
> To make package build process independent of the shell environment we
> should remove U_BOOT_CONFIG and U_BOOT_BIN passing through export call.
> So we migrate to template-based debian/rules file.
> 
> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> ---
>  .../u-boot/files/debian/{rules => rules.tmpl}          |  2 +-
>  meta/recipes-bsp/u-boot/u-boot-custom.inc              | 10 ++++------
>  2 files changed, 5 insertions(+), 7 deletions(-)
>  rename meta/recipes-bsp/u-boot/files/debian/{rules => rules.tmpl} (93%)
> 
> diff --git a/meta/recipes-bsp/u-boot/files/debian/rules b/meta/recipes-bsp/u-boot/files/debian/rules.tmpl
> similarity index 93%
> rename from meta/recipes-bsp/u-boot/files/debian/rules
> rename to meta/recipes-bsp/u-boot/files/debian/rules.tmpl
> index 85bda91..0507fc5 100755
> --- a/meta/recipes-bsp/u-boot/files/debian/rules
> +++ b/meta/recipes-bsp/u-boot/files/debian/rules.tmpl
> @@ -13,7 +13,7 @@ SET_CROSS_BUILD_TOOLS=CROSS_BUILD_TOOLS=y
>  endif
>  
>  override_dh_auto_build:
> -	$(MAKE) $(PARALLEL_MAKE) $(U_BOOT_CONFIG)
> +	$(MAKE) $(PARALLEL_MAKE) ${U_BOOT_CONFIG}
>  	$(MAKE) $(PARALLEL_MAKE) ${U_BOOT_BIN}
>  	./scripts/get_default_envs.sh >u-boot-initial-env
>  	$(MAKE) $(PARALLEL_MAKE) $(SET_CROSS_BUILD_TOOLS) NO_SDL=1 tools-only envtools
> diff --git a/meta/recipes-bsp/u-boot/u-boot-custom.inc b/meta/recipes-bsp/u-boot/u-boot-custom.inc
> index 9984d8c..819206a 100644
> --- a/meta/recipes-bsp/u-boot/u-boot-custom.inc
> +++ b/meta/recipes-bsp/u-boot/u-boot-custom.inc
> @@ -26,8 +26,8 @@ python() {
>  
>  DEBIAN_BUILD_DEPENDS ?= "bc, bison, flex, device-tree-compiler, git"
>  
> -TEMPLATE_FILES = "debian/control.tmpl"
> -TEMPLATE_VARS += "MACHINE DEBIAN_BUILD_DEPENDS"
> +TEMPLATE_FILES = "debian/control.tmpl debian/rules.tmpl"
> +TEMPLATE_VARS += "MACHINE DEBIAN_BUILD_DEPENDS U_BOOT_CONFIG U_BOOT_BIN"
>  
>  U_BOOT_TOOLS_PACKAGE ?= "0"
>  U_BOOT_CONFIG_PACKAGE ?= "0"
> @@ -70,9 +70,7 @@ u-boot-initial-env /etc
>  fw_env.config      /etc
>  EOF
>      fi
> -}
>  
> -dpkg_runbuild_prepend() {
> -    export U_BOOT_CONFIG="${U_BOOT_CONFIG}"
> -    export U_BOOT_BIN="${U_BOOT_BIN}"
> +    # restore execute permissions
> +    chmod a+x ${S}/debian/rules
>  }
> 

This will likely break users providing their own rules while expecting
the interface to stay the same. Better use the builder-export magic
consistently, not the tmpl pattern.

Comment likely applies to other recipes inc files as well.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux