[PATCH v3 00/20] Sbuild/Schroot migration

public inbox for isar-users@googlegroups.com
 help / color / mirror / Atom feed

* [PATCH v3 00/20] Sbuild/Schroot migration
@ 2021-12-08 13:05 Uladzimir Bely
  2021-12-08 13:05 ` [PATCH v3 01/20] dpkg-gbp: Use separate command to export tarball Uladzimir Bely
                   ` (20 more replies)
  0 siblings, 21 replies; 23+ messages in thread
From: Uladzimir Bely @ 2021-12-08 13:05 UTC (permalink / raw)
  To: isar-users

This is a patchset showing how sbuild/schroot tools can be integrated
into Isar build system.

Base schroot image is created with sbuild-chroot-target (or
sbuild-chroot-host in case of cross-build) recipe. These images are based
on isar-bootstrap ones, but include some build-related stuff preinstalled.

To use this changes you need to have sbuild and schroot installed and
configured. Current user should be added to sbuild group.

If 'kas-container' is used, it should be modified to support sbuild:
- /var/lib/schroot/union/overlay should be externally mounted (-v option)
- export KAS_IMAGE_VERSION=2.6.2-sbuild
- export KAS_CONTAINER_IMAGE_PATH=ghcr.io/wiselord
Here I uploaded a docker image that has preinstalled software (sbuild,
schroot, etc) required to run sbuild. Later, the changes might be
included in official kas docker images.

If 'gitlab' is used, the similar changes are required:
- 'image: ghcr.io/wiselord/kas-isar:2.6.2-sbuild' in .gitlab-ci.yml
- external dir for schroot overlay should also be specified in
/etc/gitlab-runner/config.toml:
volumes = ["/path/to/overlay:/var/lib/schroot/union/overlay"]

The patchset was tested with 'meta-iot2050' and 'xenomai-images'
downstreams. Some patches are required for these downstreams,
appropriate pull requests will be created soon.

Changes since v2:
- patches reworked/squashed for easier reading and understanding
- fixed building foreigh architectures with kas-docker
- implemented support of ccache
- fixed devshell and devshell_nodeps
Changes since v1:
- parallel builds with different BUILD_DIR are supported
- parallel multiconfig targets in one build are supported
- per-task schroot configuration in /etc/schroot/ is now used
- patchset now passes Jenkins CI (so patches changes RFC => PATCH)

Current limitations:
- parallel building of several packages requires more free space in
comparison with buildchroot-based architecture. This happens due
the sbuild architecture which uses some basic layer (common for all
packages) and per-package separate layer (where builddeps are downloaded
and installed).

For experiments, the code can be taken from Isar 'ubely/sbuild' branch.
This is a dev branch, so sometimes it can be force-pushed when some
fixes are applied of rebased to 'next' branch.

First 9 patches (1..9) - are just preparation patches. With them, Isar will
continue using buildchroot-based architecture. So, potentially,
they can be merged as a subseries.

Remaining 11 patches (10..20) represent a sbuild-related changes. 

Anton Mikanovich (10):
  template: Make templates passthrough
  linux-module: Do not use shell environment
  u-boot: Do not use shell environment
  trusted-firmware: Do not use shell environment
  optee-os: Do not use shell environment
  kselftest: Do not use shell environment
  linux-mainline: Move cfg fragment test to debian/rules
  dpkg: Build packages with sbuild
  sbuild: Introduce environment variables export API
  linux-custom: Prepare kernel config inside sbuild

Uladzimir Bely (10):
  dpkg-gbp: Use separate command to export tarball
  dpkg-gbp: Use host tools for dsc preparation
  sbuild: Add recipes for host and target rootfs to run sbuild
  sbuild: Introduce a class for another build method
  dpkg: Remove builddeps install task.
  dpkg-base: Switch devshell to use schroot
  dpkg-base: Switch apt_fetch and apt_unpack to use schroot
  dpkg-base: Cleanup from buildchroot parts.
  doc: Add sbuild-related documentation
  sbuild: add ccache support

 doc/user_manual.md                            |  22 ++-
 .../optee-os/optee-os-stm32mp15x_3.11.0.bb    |   2 +-
 .../linux/linux-mainline_5.4.70.bb            |  16 +-
 meta/classes/dpkg-base.bbclass                |  98 ++++------
 meta/classes/dpkg-gbp.bbclass                 |  26 +--
 meta/classes/dpkg.bbclass                     |  53 ++---
 meta/classes/sbuild.bbclass                   | 184 ++++++++++++++++++
 meta/classes/template.bbclass                 |   7 +-
 meta/conf/bitbake.conf                        |   2 +
 .../files/debian/{rules => rules.tmpl}        |   4 +-
 meta/recipes-bsp/optee-os/optee-os-custom.inc |   9 +-
 .../files/debian/{rules => rules.tmpl}        |   4 +-
 .../trusted-firmware-a-custom.inc             |   9 +-
 .../u-boot/files/debian/{rules => rules.tmpl} |   2 +-
 meta/recipes-bsp/u-boot/u-boot-custom.inc     |   9 +-
 .../sbuild-chroot/sbuild-chroot-host.bb       |  13 ++
 .../sbuild-chroot/sbuild-chroot-target.bb     |  10 +
 .../sbuild-chroot/sbuild-chroot.inc           |  39 ++++
 .../kselftest/files/{rules => rules.tmpl}     |   0
 meta/recipes-kernel/kselftest/kselftest.inc   |  21 +-
 .../linux-module/files/debian/rules           |  37 ----
 .../linux-module/files/debian/rules.tmpl      |  52 +++++
 meta/recipes-kernel/linux-module/module.inc   |  19 +-
 meta/recipes-kernel/linux/linux-custom.inc    |   7 +-
 24 files changed, 437 insertions(+), 208 deletions(-)
 create mode 100644 meta/classes/sbuild.bbclass
 rename meta/recipes-bsp/optee-os/files/debian/{rules => rules.tmpl} (75%)
 rename meta/recipes-bsp/trusted-firmware-a/files/debian/{rules => rules.tmpl} (77%)
 rename meta/recipes-bsp/u-boot/files/debian/{rules => rules.tmpl} (94%)
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
 rename meta/recipes-kernel/kselftest/files/{rules => rules.tmpl} (100%)
 delete mode 100755 meta/recipes-kernel/linux-module/files/debian/rules
 create mode 100755 meta/recipes-kernel/linux-module/files/debian/rules.tmpl

-- 
2.20.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* [PATCH v3 01/20] dpkg-gbp: Use separate command to export tarball
  2021-12-08 13:05 [PATCH v3 00/20] Sbuild/Schroot migration Uladzimir Bely
@ 2021-12-08 13:05 ` Uladzimir Bely
  2021-12-08 13:05 ` [PATCH v3 02/20] template: Make templates passthrough Uladzimir Bely
                   ` (19 subsequent siblings)
  20 siblings, 0 replies; 23+ messages in thread
From: Uladzimir Bely @ 2021-12-08 13:05 UTC (permalink / raw)
  To: isar-users

We don't actually build the package with gbp, but only prepare it for
the building with external dpkg-buildpackage command. In case there is
no need in real build we can perform only export which will produce
upstream tarball to pass for later building.
This allows to use any build tool for later processing.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-gbp.bbclass | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index d956e8c3..ddf23ca5 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -30,5 +30,9 @@ do_install_builddeps_append() {
 }
 
 dpkg_runbuild_prepend() {
-    export GBP_PREFIX="gbp buildpackage --git-ignore-new ${GBP_EXTRA_OPTIONS} --git-builder="
+    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+        sh -c "cd ${PP}/${PPS} && gbp buildpackage --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}"
+    # NOTE: `buildpackage --git-builder=/bin/true --git-pristine-tar` is used
+    # for compatibility with gbp version froms debian-stretch. In newer distros
+    # it's possible to use a subcommand `export-orig --pristine-tar`
 }
-- 
2.20.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* [PATCH v3 02/20] template: Make templates passthrough
  2021-12-08 13:05 [PATCH v3 00/20] Sbuild/Schroot migration Uladzimir Bely
  2021-12-08 13:05 ` [PATCH v3 01/20] dpkg-gbp: Use separate command to export tarball Uladzimir Bely
@ 2021-12-08 13:05 ` Uladzimir Bely
  2021-12-08 13:05 ` [PATCH v3 03/20] linux-module: Do not use shell environment Uladzimir Bely
                   ` (18 subsequent siblings)
  20 siblings, 0 replies; 23+ messages in thread
From: Uladzimir Bely @ 2021-12-08 13:05 UTC (permalink / raw)
  To: isar-users

From: Anton Mikanovich <amikan@ilbers.de>

There is no need to store original template files after the conversion
in real scenarios. To make working folder little cleaner we can remove
them. This can be disabled with TEMPLATE_FILES_KEEP variable.
Moreover output file should have exactly the same flags as input one,
which is usefull for the cases like debian/rules or other executables.
So we should copy this part of metadata after the conversion.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/template.bbclass              | 7 ++++++-
 meta/recipes-kernel/linux/linux-custom.inc | 3 ---
 2 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/meta/classes/template.bbclass b/meta/classes/template.bbclass
index fb9d1186..f5760e15 100644
--- a/meta/classes/template.bbclass
+++ b/meta/classes/template.bbclass
@@ -4,11 +4,12 @@
 # SPDX-License-Identifier: MIT
 
 TEMPLATE_FILES ?= ""
+TEMPLATE_FILES_KEEP ?= "0"
 TEMPLATE_VARS ?= "PN PV DESCRIPTION HOMEPAGE MAINTAINER DISTRO_ARCH"
 
 do_transform_template[vardeps] = "TEMPLATE_FILES ${TEMPLATE_VARS}"
 python do_transform_template() {
-    import subprocess, contextlib
+    import subprocess, contextlib, shutil
 
     workdir = os.path.normpath(d.getVar('WORKDIR', True))
 
@@ -56,5 +57,9 @@ python do_transform_template() {
                                           stdout=output, env=env))
             if process.wait() != 0:
                 bb.fatal("processing of template failed")
+
+        shutil.copymode(template_file, output_file)
+        if d.getVar('TEMPLATE_FILES_KEEP', True) != '1':
+            os.remove(template_file)
 }
 addtask do_transform_template after do_unpack
diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index ed89aa09..57740860 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -122,9 +122,6 @@ do_prepare_build_prepend() {
 	rm -rf ${S}/debian
 	cp -r ${WORKDIR}/debian ${S}/
 
-	# remove templates from the source tree
-	find ${S}/debian -name *.tmpl | xargs rm -f
-
 	# rename install/remove hooks to match user-specified name for our linux-image package
 	mv ${S}/debian/linux-image.postinst ${S}/debian/linux-image-${KERNEL_NAME_PROVIDED}.postinst
 	mv ${S}/debian/linux-image.postrm ${S}/debian/linux-image-${KERNEL_NAME_PROVIDED}.postrm
-- 
2.20.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* [PATCH v3 03/20] linux-module: Do not use shell environment
  2021-12-08 13:05 [PATCH v3 00/20] Sbuild/Schroot migration Uladzimir Bely
  2021-12-08 13:05 ` [PATCH v3 01/20] dpkg-gbp: Use separate command to export tarball Uladzimir Bely
  2021-12-08 13:05 ` [PATCH v3 02/20] template: Make templates passthrough Uladzimir Bely
@ 2021-12-08 13:05 ` Uladzimir Bely
  2021-12-08 13:05 ` [PATCH v3 04/20] u-boot: " Uladzimir Bely
                   ` (17 subsequent siblings)
  20 siblings, 0 replies; 23+ messages in thread
From: Uladzimir Bely @ 2021-12-08 13:05 UTC (permalink / raw)
  To: isar-users

From: Anton Mikanovich <amikan@ilbers.de>

To make package build process independent of the shell environment we
should remove KDIR and PN passing through export call.
KDIR can be prepared during package build. This also will allow not to
rely on previous builddeps install task.
To pass PN variable we can just migrate to template-based debian/rules
file.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 .../linux-module/files/debian/rules           | 37 -------------
 .../linux-module/files/debian/rules.tmpl      | 52 +++++++++++++++++++
 meta/recipes-kernel/linux-module/module.inc   | 19 ++-----
 3 files changed, 55 insertions(+), 53 deletions(-)
 delete mode 100755 meta/recipes-kernel/linux-module/files/debian/rules
 create mode 100755 meta/recipes-kernel/linux-module/files/debian/rules.tmpl

diff --git a/meta/recipes-kernel/linux-module/files/debian/rules b/meta/recipes-kernel/linux-module/files/debian/rules
deleted file mode 100755
index 59720b37..00000000
--- a/meta/recipes-kernel/linux-module/files/debian/rules
+++ /dev/null
@@ -1,37 +0,0 @@
-#!/usr/bin/make -f
-
-# Debian rules for custom kernel module build
-#
-# This software is a part of ISAR.
-# Copyright (c) Siemens AG, 2018
-#
-# SPDX-License-Identifier: MIT
-
-export DEB_BUILD_OPTIONS=parallel=$(shell nproc)
-
-export CROSS_COMPILE=$(DEB_HOST_GNU_TYPE)-
-
-ifeq ($(DEB_HOST_GNU_CPU), arm)
-export ARCH=arm
-endif
-ifeq ($(DEB_HOST_GNU_CPU), aarch64)
-export ARCH=arm64
-endif
-ifeq ($(DEB_HOST_GNU_CPU), riscv64)
-export ARCH=riscv
-endif
-ifneq (,$(findstring 86,$(DEB_HOST_GNU_CPU)))
-export ARCH=x86
-endif
-
-override_dh_auto_clean:
-	$(MAKE) -C $(KDIR) M=$(PWD) clean
-
-override_dh_auto_build:
-	$(MAKE) -C $(KDIR) M=$(PWD) modules
-
-override_dh_auto_install:
-	$(MAKE) -C $(KDIR) M=$(PWD) INSTALL_MOD_PATH=$(PWD)/debian/$(PN) modules_install
-
-%:
-	CFLAGS= LDFLAGS= dh $@ --parallel
diff --git a/meta/recipes-kernel/linux-module/files/debian/rules.tmpl b/meta/recipes-kernel/linux-module/files/debian/rules.tmpl
new file mode 100755
index 00000000..52453e5f
--- /dev/null
+++ b/meta/recipes-kernel/linux-module/files/debian/rules.tmpl
@@ -0,0 +1,52 @@
+#!/usr/bin/make -f
+
+# Debian rules for custom kernel module build
+#
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2018
+#
+# SPDX-License-Identifier: MIT
+
+export DEB_BUILD_OPTIONS=parallel=$(shell nproc)
+
+export CROSS_COMPILE=$(DEB_HOST_GNU_TYPE)-
+
+ifeq ($(DEB_HOST_GNU_CPU), arm)
+export ARCH=arm
+endif
+ifeq ($(DEB_HOST_GNU_CPU), aarch64)
+export ARCH=arm64
+endif
+ifeq ($(DEB_HOST_GNU_CPU), riscv64)
+export ARCH=riscv
+endif
+ifneq (,$(findstring 86,$(DEB_HOST_GNU_CPU)))
+export ARCH=x86
+endif
+
+# Custom kernels contain the build folder directly.
+KDIR := $(shell dpkg -L linux-headers-${KERNEL_NAME} | grep "/lib/modules/.*/build")
+ifeq ($(KDIR),)
+# Debian kernels install that folder indirectly via a dependency.
+KERNEL_DEP := $(shell dpkg-query -W -f '$${Depends}' linux-headers-${KERNEL_NAME} | sed 's/.*\(linux-headers-[[:graph:]]*\).*/\1/')
+KDIR := $(shell dpkg -L $(KERNEL_DEP) | grep "/lib/modules/.*/build")
+endif
+
+# With some build systems like sbuild `dh clean` can be called twice:
+# first, by build system itself before chroot's apt database updated,
+# second, by dpkg-buildpackage during build. So, KDIR may be empty
+# in first case (while no dependencies are yet installed), and a broken
+# 'make ... clean' command is produced. Just skip override in this case.
+ifneq ($(KDIR),)
+override_dh_auto_clean:
+	$(MAKE) -C $(KDIR) M=$(PWD) clean
+endif
+
+override_dh_auto_build:
+	$(MAKE) -C $(KDIR) M=$(PWD) modules
+
+override_dh_auto_install:
+	$(MAKE) -C $(KDIR) M=$(PWD) INSTALL_MOD_PATH=$(PWD)/debian/${PN} modules_install
+
+%:
+	CFLAGS= LDFLAGS= dh $@ --parallel
diff --git a/meta/recipes-kernel/linux-module/module.inc b/meta/recipes-kernel/linux-module/module.inc
index 0515621a..9ae3af9b 100644
--- a/meta/recipes-kernel/linux-module/module.inc
+++ b/meta/recipes-kernel/linux-module/module.inc
@@ -22,8 +22,9 @@ AUTOLOAD ?= ""
 inherit dpkg
 
 TEMPLATE_FILES = "debian/control.tmpl \
-                  debian/changelog.tmpl"
-TEMPLATE_VARS += "KERNEL_NAME"
+                  debian/changelog.tmpl \
+                  debian/rules.tmpl"
+TEMPLATE_VARS += "KERNEL_NAME PN"
 
 do_prepare_build() {
     cp -r ${WORKDIR}/debian ${S}/
@@ -32,17 +33,3 @@ do_prepare_build() {
         echo "echo $module >> /etc/modules" >> ${S}/debian/postinst
     done
 }
-
-dpkg_runbuild_prepend() {
-    # Custom kernels contain the build folder directly.
-    export KDIR=$(dpkg -L --root=${BUILDCHROOT_DIR} linux-headers-${KERNEL_NAME} | \
-                  grep "/lib/modules/.*/build")
-    if [ -z "$KDIR" ]; then
-        # Debian kernels install that folder indirectly via a dependency.
-        KERNEL_DEP=$(dpkg-query -W -f '${Depends}' --admindir=${BUILDCHROOT_DIR}/var/lib/dpkg \
-                     linux-headers-${KERNEL_NAME} | sed 's/.*\(linux-headers-[[:graph:]]*\).*/\1/')
-        export KDIR=$(dpkg -L --root=${BUILDCHROOT_DIR} ${KERNEL_DEP} | \
-                      grep "/lib/modules/.*/build")
-    fi
-    export PN=${PN}
-}
-- 
2.20.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* [PATCH v3 04/20] u-boot: Do not use shell environment
  2021-12-08 13:05 [PATCH v3 00/20] Sbuild/Schroot migration Uladzimir Bely
                   ` (2 preceding siblings ...)
  2021-12-08 13:05 ` [PATCH v3 03/20] linux-module: Do not use shell environment Uladzimir Bely
@ 2021-12-08 13:05 ` Uladzimir Bely
  2021-12-08 13:05 ` [PATCH v3 05/20] trusted-firmware: " Uladzimir Bely
                   ` (16 subsequent siblings)
  20 siblings, 0 replies; 23+ messages in thread
From: Uladzimir Bely @ 2021-12-08 13:05 UTC (permalink / raw)
  To: isar-users

From: Anton Mikanovich <amikan@ilbers.de>

To make package build process independent of the shell environment we
should remove U_BOOT_CONFIG and U_BOOT_BIN passing through export call.
So we migrate to template-based debian/rules file.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 .../u-boot/files/debian/{rules => rules.tmpl}            | 2 +-
 meta/recipes-bsp/u-boot/u-boot-custom.inc                | 9 ++-------
 2 files changed, 3 insertions(+), 8 deletions(-)
 rename meta/recipes-bsp/u-boot/files/debian/{rules => rules.tmpl} (94%)

diff --git a/meta/recipes-bsp/u-boot/files/debian/rules b/meta/recipes-bsp/u-boot/files/debian/rules.tmpl
similarity index 94%
rename from meta/recipes-bsp/u-boot/files/debian/rules
rename to meta/recipes-bsp/u-boot/files/debian/rules.tmpl
index 3d667620..806b01fe 100755
--- a/meta/recipes-bsp/u-boot/files/debian/rules
+++ b/meta/recipes-bsp/u-boot/files/debian/rules.tmpl
@@ -13,7 +13,7 @@ SET_CROSS_BUILD_TOOLS=CROSS_BUILD_TOOLS=y
 endif
 
 override_dh_auto_build:
-	$(MAKE) $(PARALLEL_MAKE) $(U_BOOT_CONFIG)
+	$(MAKE) $(PARALLEL_MAKE) ${U_BOOT_CONFIG}
 	$(MAKE) $(PARALLEL_MAKE) ${U_BOOT_BIN}
 	$(MAKE) -n u-boot-initial-env >/dev/null 2>&1; if [ $$? -ne 2 ]; then \
 		$(MAKE) $(PARALLEL_MAKE) u-boot-initial-env; \
diff --git a/meta/recipes-bsp/u-boot/u-boot-custom.inc b/meta/recipes-bsp/u-boot/u-boot-custom.inc
index 9984d8cc..2af7ada1 100644
--- a/meta/recipes-bsp/u-boot/u-boot-custom.inc
+++ b/meta/recipes-bsp/u-boot/u-boot-custom.inc
@@ -26,8 +26,8 @@ python() {
 
 DEBIAN_BUILD_DEPENDS ?= "bc, bison, flex, device-tree-compiler, git"
 
-TEMPLATE_FILES = "debian/control.tmpl"
-TEMPLATE_VARS += "MACHINE DEBIAN_BUILD_DEPENDS"
+TEMPLATE_FILES = "debian/control.tmpl debian/rules.tmpl"
+TEMPLATE_VARS += "MACHINE DEBIAN_BUILD_DEPENDS U_BOOT_CONFIG U_BOOT_BIN"
 
 U_BOOT_TOOLS_PACKAGE ?= "0"
 U_BOOT_CONFIG_PACKAGE ?= "0"
@@ -71,8 +71,3 @@ fw_env.config      /etc
 EOF
     fi
 }
-
-dpkg_runbuild_prepend() {
-    export U_BOOT_CONFIG="${U_BOOT_CONFIG}"
-    export U_BOOT_BIN="${U_BOOT_BIN}"
-}
-- 
2.20.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* [PATCH v3 05/20] trusted-firmware: Do not use shell environment
  2021-12-08 13:05 [PATCH v3 00/20] Sbuild/Schroot migration Uladzimir Bely
                   ` (3 preceding siblings ...)
  2021-12-08 13:05 ` [PATCH v3 04/20] u-boot: " Uladzimir Bely
@ 2021-12-08 13:05 ` Uladzimir Bely
  2021-12-08 13:05 ` [PATCH v3 06/20] optee-os: " Uladzimir Bely
                   ` (15 subsequent siblings)
  20 siblings, 0 replies; 23+ messages in thread
From: Uladzimir Bely @ 2021-12-08 13:05 UTC (permalink / raw)
  To: isar-users

From: Anton Mikanovich <amikan@ilbers.de>

To make package build process independent of the shell environment we
should remove TF_A_PLATFORM and TF_A_EXTRA_BUILDARGS passing through
export call. So we migrate to template-based debian/rules file.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 .../files/debian/{rules => rules.tmpl}                   | 4 ++--
 .../trusted-firmware-a/trusted-firmware-a-custom.inc     | 9 ++-------
 2 files changed, 4 insertions(+), 9 deletions(-)
 rename meta/recipes-bsp/trusted-firmware-a/files/debian/{rules => rules.tmpl} (77%)

diff --git a/meta/recipes-bsp/trusted-firmware-a/files/debian/rules b/meta/recipes-bsp/trusted-firmware-a/files/debian/rules.tmpl
similarity index 77%
rename from meta/recipes-bsp/trusted-firmware-a/files/debian/rules
rename to meta/recipes-bsp/trusted-firmware-a/files/debian/rules.tmpl
index 70e1dd66..6dbf44db 100755
--- a/meta/recipes-bsp/trusted-firmware-a/files/debian/rules
+++ b/meta/recipes-bsp/trusted-firmware-a/files/debian/rules.tmpl
@@ -12,8 +12,8 @@ export CROSS_COMPILE=$(DEB_HOST_GNU_TYPE)-
 endif
 
 override_dh_auto_build:
-	CFLAGS= LDFLAGS= $(MAKE) $(PARALLEL_MAKE) PLAT=$(TF_A_PLATFORM) \
-		$(TF_A_EXTRA_BUILDARGS)
+	CFLAGS= LDFLAGS= $(MAKE) $(PARALLEL_MAKE) PLAT=${TF_A_PLATFORM} \
+		${TF_A_EXTRA_BUILDARGS}
 
 %:
 	dh $@
diff --git a/meta/recipes-bsp/trusted-firmware-a/trusted-firmware-a-custom.inc b/meta/recipes-bsp/trusted-firmware-a/trusted-firmware-a-custom.inc
index 1142bfce..64153c6b 100644
--- a/meta/recipes-bsp/trusted-firmware-a/trusted-firmware-a-custom.inc
+++ b/meta/recipes-bsp/trusted-firmware-a/trusted-firmware-a-custom.inc
@@ -22,8 +22,8 @@ DEBIAN_BUILD_DEPENDS ?= ""
 
 PROVIDES += "trusted-firmware-a-${TF_A_NAME}"
 
-TEMPLATE_FILES = "debian/control.tmpl"
-TEMPLATE_VARS += "TF_A_NAME DEBIAN_BUILD_DEPENDS"
+TEMPLATE_FILES = "debian/control.tmpl debian/rules.tmpl"
+TEMPLATE_VARS += "TF_A_NAME DEBIAN_BUILD_DEPENDS TF_A_PLATFORM TF_A_EXTRA_BUILDARGS"
 
 do_prepare_build() {
     cp -r ${WORKDIR}/debian ${S}/
@@ -36,8 +36,3 @@ do_prepare_build() {
             ${S}/debian/trusted-firmware-a-${TF_A_NAME}.install
     done
 }
-
-dpkg_runbuild_prepend() {
-    export TF_A_PLATFORM="${TF_A_PLATFORM}"
-    export TF_A_EXTRA_BUILDARGS="${TF_A_EXTRA_BUILDARGS}"
-}
-- 
2.20.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* [PATCH v3 06/20] optee-os: Do not use shell environment
  2021-12-08 13:05 [PATCH v3 00/20] Sbuild/Schroot migration Uladzimir Bely
                   ` (4 preceding siblings ...)
  2021-12-08 13:05 ` [PATCH v3 05/20] trusted-firmware: " Uladzimir Bely
@ 2021-12-08 13:05 ` Uladzimir Bely
  2021-12-08 13:05 ` [PATCH v3 07/20] kselftest: " Uladzimir Bely
                   ` (14 subsequent siblings)
  20 siblings, 0 replies; 23+ messages in thread
From: Uladzimir Bely @ 2021-12-08 13:05 UTC (permalink / raw)
  To: isar-users

From: Anton Mikanovich <amikan@ilbers.de>

To make package build process independent of the shell environment we
should remove OPTEE_PLATFORM and OPTEE_EXTRA_BUILDARGS passing through
export call. So we migrate to template-based debian/rules file.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 .../optee-os/files/debian/{rules => rules.tmpl}          | 4 ++--
 meta/recipes-bsp/optee-os/optee-os-custom.inc            | 9 ++-------
 2 files changed, 4 insertions(+), 9 deletions(-)
 rename meta/recipes-bsp/optee-os/files/debian/{rules => rules.tmpl} (75%)

diff --git a/meta/recipes-bsp/optee-os/files/debian/rules b/meta/recipes-bsp/optee-os/files/debian/rules.tmpl
similarity index 75%
rename from meta/recipes-bsp/optee-os/files/debian/rules
rename to meta/recipes-bsp/optee-os/files/debian/rules.tmpl
index d2e9900f..9ab80dfe 100755
--- a/meta/recipes-bsp/optee-os/files/debian/rules
+++ b/meta/recipes-bsp/optee-os/files/debian/rules.tmpl
@@ -12,8 +12,8 @@ export CROSS_COMPILE=$(DEB_HOST_GNU_TYPE)-
 endif
 
 override_dh_auto_build:
-	CFLAGS= LDFLAGS= $(MAKE) $(PARALLEL_MAKE) PLATFORM=$(OPTEE_PLATFORM) \
-		$(OPTEE_EXTRA_BUILDARGS)
+	CFLAGS= LDFLAGS= $(MAKE) $(PARALLEL_MAKE) PLATFORM=${OPTEE_PLATFORM} \
+		${OPTEE_EXTRA_BUILDARGS}
 
 %:
 	dh $@
diff --git a/meta/recipes-bsp/optee-os/optee-os-custom.inc b/meta/recipes-bsp/optee-os/optee-os-custom.inc
index 1bd51969..23612d81 100644
--- a/meta/recipes-bsp/optee-os/optee-os-custom.inc
+++ b/meta/recipes-bsp/optee-os/optee-os-custom.inc
@@ -22,8 +22,8 @@ DEBIAN_BUILD_DEPENDS ?= "python3-pycryptodome:native, python3-pyelftools"
 
 PROVIDES += "optee-os-${OPTEE_NAME}"
 
-TEMPLATE_FILES = "debian/control.tmpl"
-TEMPLATE_VARS += "OPTEE_NAME DEBIAN_BUILD_DEPENDS"
+TEMPLATE_FILES = "debian/control.tmpl debian/rules.tmpl"
+TEMPLATE_VARS += "OPTEE_NAME DEBIAN_BUILD_DEPENDS OPTEE_PLATFORM OPTEE_EXTRA_BUILDARGS"
 
 # split strip platform flavor, if any, from the specified platform string
 OPTEE_PLATFORM_BASE = "${@d.getVar('OPTEE_PLATFORM').split('-')[0]}"
@@ -39,8 +39,3 @@ do_prepare_build() {
             ${S}/debian/optee-os-${OPTEE_NAME}.install
     done
 }
-
-dpkg_runbuild_prepend() {
-    export OPTEE_PLATFORM="${OPTEE_PLATFORM}"
-    export OPTEE_EXTRA_BUILDARGS="${OPTEE_EXTRA_BUILDARGS}"
-}
-- 
2.20.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* [PATCH v3 07/20] kselftest: Do not use shell environment
  2021-12-08 13:05 [PATCH v3 00/20] Sbuild/Schroot migration Uladzimir Bely
                   ` (5 preceding siblings ...)
  2021-12-08 13:05 ` [PATCH v3 06/20] optee-os: " Uladzimir Bely
@ 2021-12-08 13:05 ` Uladzimir Bely
  2021-12-08 13:05 ` [PATCH v3 08/20] linux-mainline: Move cfg fragment test to debian/rules Uladzimir Bely
                   ` (13 subsequent siblings)
  20 siblings, 0 replies; 23+ messages in thread
From: Uladzimir Bely @ 2021-12-08 13:05 UTC (permalink / raw)
  To: isar-users

From: Anton Mikanovich <amikan@ilbers.de>

To make package build process independent of the shell environment we
should remove KSELFTEST_ARGS passing through export call. The same
logic can be done by internal recipe parser and then pass variable to
template-based debian/rules file.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 .../kselftest/files/{rules => rules.tmpl}     |  0
 meta/recipes-kernel/kselftest/kselftest.inc   | 21 +++++++------------
 2 files changed, 8 insertions(+), 13 deletions(-)
 rename meta/recipes-kernel/kselftest/files/{rules => rules.tmpl} (100%)

diff --git a/meta/recipes-kernel/kselftest/files/rules b/meta/recipes-kernel/kselftest/files/rules.tmpl
similarity index 100%
rename from meta/recipes-kernel/kselftest/files/rules
rename to meta/recipes-kernel/kselftest/files/rules.tmpl
diff --git a/meta/recipes-kernel/kselftest/kselftest.inc b/meta/recipes-kernel/kselftest/kselftest.inc
index 2a76028b..6187d8e4 100644
--- a/meta/recipes-kernel/kselftest/kselftest.inc
+++ b/meta/recipes-kernel/kselftest/kselftest.inc
@@ -33,26 +33,21 @@ DEBIAN_BUILD_DEPENDS ?= " \
     llvm:native, \
 "
 
-SRC_URI += "file://rules"
+SRC_URI += "file://rules.tmpl"
 S = "${WORKDIR}/linux-${PV}"
 
+TEMPLATE_FILES = "rules.tmpl"
+TEMPLATE_VARS += "KSELFTEST_ARGS"
+
 KSELFTEST_TARGETS ?= ""
 KSELFTEST_SKIP_TARGETS ?= ""
 KSELFTEST_FORCE_TARGETS ?= "0"
 
+KSELFTEST_ARGS = "${@ "TARGETS=\"${KSELFTEST_TARGETS}\"" if d.getVar('KSELFTEST_TARGETS', True) else ''}"
+KSELFTEST_ARGS_append = "${@ " FORCE_TARGETS=1" if d.getVar('KSELFTEST_FORCE_TARGETS', True) == '1' else ''}"
+KSELFTEST_ARGS_append .= "${@ " SKIP_TARGETS=\"${KSELFTEST_SKIP_TARGETS}\"" if d.getVar('KSELFTEST_SKIP_TARGETS', True) else ''}"
+
 do_prepare_build[cleandirs] += "${S}/debian"
 do_prepare_build() {
     deb_debianize
 }
-
-dpkg_runbuild_prepend() {
-    if [ -n "${KSELFTEST_TARGETS}" ];then
-        export KSELFTEST_ARGS="TARGETS=\"${KSELFTEST_TARGETS}\""
-    fi
-    if [ ${KSELFTEST_FORCE_TARGETS} -eq 1 ];then
-        export KSELFTEST_ARGS="${KSELFTEST_ARGS} FORCE_TARGETS=1"
-    fi
-    if [ -n "${KSELFTEST_SKIP_TARGETS}" ];then
-        export KSELFTEST_ARGS="${KSELFTEST_ARGS} SKIP_TARGETS=\"${KSELFTEST_SKIP_TARGETS}\""
-    fi
-}
-- 
2.20.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* [PATCH v3 08/20] linux-mainline: Move cfg fragment test to debian/rules
  2021-12-08 13:05 [PATCH v3 00/20] Sbuild/Schroot migration Uladzimir Bely
                   ` (6 preceding siblings ...)
  2021-12-08 13:05 ` [PATCH v3 07/20] kselftest: " Uladzimir Bely
@ 2021-12-08 13:05 ` Uladzimir Bely
  2021-12-08 13:05 ` [PATCH v3 09/20] dpkg-gbp: Use host tools for dsc preparation Uladzimir Bely
                   ` (12 subsequent siblings)
  20 siblings, 0 replies; 23+ messages in thread
From: Uladzimir Bely @ 2021-12-08 13:05 UTC (permalink / raw)
  To: isar-users

From: Anton Mikanovich <amikan@ilbers.de>

Perform all config fragments checking in Debian way.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 .../linux/linux-mainline_5.4.70.bb               | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)

diff --git a/meta-isar/recipes-kernel/linux/linux-mainline_5.4.70.bb b/meta-isar/recipes-kernel/linux/linux-mainline_5.4.70.bb
index 980e137b..28e51c0d 100644
--- a/meta-isar/recipes-kernel/linux/linux-mainline_5.4.70.bb
+++ b/meta-isar/recipes-kernel/linux/linux-mainline_5.4.70.bb
@@ -25,11 +25,13 @@ LINUX_VERSION_EXTENSION = "-isar"
 
 # For testing purposes only
 dpkg_configure_kernel_append() {
-    if ! grep "# CONFIG_MTD is not set" ${S}/${KERNEL_BUILD_DIR}/.config && \
-       ! grep "# CONFIG_MTD_UBI is not set" ${S}/${KERNEL_BUILD_DIR}/.config; then
-        grep "# CONFIG_UBIFS_FS is not set" ${S}/${KERNEL_BUILD_DIR}/.config || \
-            bbfatal "Self-check failed: CONFIG_UBIFS_FS still enabled"
-    fi
-    grep "CONFIG_ROOT_NFS=y" ${S}/${KERNEL_BUILD_DIR}/.config || \
-        bbfatal "Self-check failed: CONFIG_ROOT_NFS not enabled"
+cat << EOF | sed -i '/^override_dh_auto_build/ r /dev/stdin' ${S}/debian/rules
+	if ! grep "# CONFIG_MTD is not set" \$(O)/.config && \\
+	   ! grep "# CONFIG_MTD_UBI is not set" \$(O)/.config; then \\
+	    grep "# CONFIG_UBIFS_FS is not set" \$(O)/.config || \\
+	        (echo "Self-check failed: CONFIG_UBIFS_FS still enabled" && exit 1); \\
+	fi
+	grep "CONFIG_ROOT_NFS=y" \$(O)/.config || \\
+	    (echo "Self-check failed: CONFIG_ROOT_NFS not enabled" && exit 1)
+EOF
 }
-- 
2.20.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* [PATCH v3 09/20] dpkg-gbp: Use host tools for dsc preparation
  2021-12-08 13:05 [PATCH v3 00/20] Sbuild/Schroot migration Uladzimir Bely
                   ` (7 preceding siblings ...)
  2021-12-08 13:05 ` [PATCH v3 08/20] linux-mainline: Move cfg fragment test to debian/rules Uladzimir Bely
@ 2021-12-08 13:05 ` Uladzimir Bely
  2021-12-08 13:05 ` [PATCH v3 10/20] sbuild: Add recipes for host and target rootfs to run sbuild Uladzimir Bely
                   ` (11 subsequent siblings)
  20 siblings, 0 replies; 23+ messages in thread
From: Uladzimir Bely @ 2021-12-08 13:05 UTC (permalink / raw)
  To: isar-users

Instead of preinstalling gbp-related packages in sbuild chroot
we will use their host versions in order to keep sbuild environment
as clean as possible.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 doc/user_manual.md            |  4 +++-
 meta/classes/dpkg-gbp.bbclass | 24 ++++--------------------
 2 files changed, 7 insertions(+), 21 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index c81c6e46..9017dd89 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -79,7 +79,9 @@ apt install \
   qemu \
   qemu-user-static \
   reprepro \
-  sudo
+  sudo \
+  git-buildpackage \
+  pristine-tar
 ```
 
 If your host is >= buster, also install the following package.
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index ddf23ca5..2fc37b2f 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -9,29 +9,13 @@ S = "${WORKDIR}/git"
 
 PATCHTOOL ?= "git"
 
-GBP_DEPENDS ?= "git-buildpackage pristine-tar"
 GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
 
-do_install_builddeps_append() {
-    dpkg_do_mounts
-    distro="${DISTRO}"
-    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
-       distro="${HOST_DISTRO}"
-    fi
-    deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} \
-        apt-get install -y -o Debug::pkgProblemResolver=yes \
-                        --no-install-recommends --download-only ${GBP_DEPENDS}
-    deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} \
-        apt-get install -y -o Debug::pkgProblemResolver=yes \
-                        --no-install-recommends ${GBP_DEPENDS}
-    dpkg_undo_mounts
-}
-
 dpkg_runbuild_prepend() {
-    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
-        sh -c "cd ${PP}/${PPS} && gbp buildpackage --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}"
+    sh -c "
+        cd ${WORKDIR}/${PPS}
+        gbp buildpackage --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}
+    "
     # NOTE: `buildpackage --git-builder=/bin/true --git-pristine-tar` is used
     # for compatibility with gbp version froms debian-stretch. In newer distros
     # it's possible to use a subcommand `export-orig --pristine-tar`
-- 
2.20.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* [PATCH v3 10/20] sbuild: Add recipes for host and target rootfs to run sbuild
  2021-12-08 13:05 [PATCH v3 00/20] Sbuild/Schroot migration Uladzimir Bely
                   ` (8 preceding siblings ...)
  2021-12-08 13:05 ` [PATCH v3 09/20] dpkg-gbp: Use host tools for dsc preparation Uladzimir Bely
@ 2021-12-08 13:05 ` Uladzimir Bely
  2021-12-08 13:05 ` [PATCH v3 11/20] sbuild: Introduce a class for another build method Uladzimir Bely
                   ` (10 subsequent siblings)
  20 siblings, 0 replies; 23+ messages in thread
From: Uladzimir Bely @ 2021-12-08 13:05 UTC (permalink / raw)
  To: isar-users

Similar to buildchroot, we need a separate rootfs to be used
for schroot.

It's based on bootstrapped rootfs, but includes several
common build-related packages.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 .../sbuild-chroot/sbuild-chroot-host.bb       | 13 +++++++
 .../sbuild-chroot/sbuild-chroot-target.bb     | 10 ++++++
 .../sbuild-chroot/sbuild-chroot.inc           | 36 +++++++++++++++++++
 3 files changed, 59 insertions(+)
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
 create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc

diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
new file mode 100644
index 00000000..aa82846b
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
@@ -0,0 +1,13 @@
+# Root filesystem for packages building
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2021 ilbers GmbH
+
+DESCRIPTION = "Isar sbuild/schroot filesystem for host"
+
+SBUILD_VARIANT = "host"
+
+require sbuild-chroot.inc
+
+ROOTFS_ARCH = "${HOST_ARCH}"
+ROOTFS_DISTRO = "${HOST_DISTRO}"
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
new file mode 100644
index 00000000..d75d783b
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
@@ -0,0 +1,10 @@
+# Root filesystem for packages building
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2021 ilbers GmbH
+
+DESCRIPTION = "Isar sbuild/schroot filesystem for target"
+
+SBUILD_VARIANT = "target"
+
+require sbuild-chroot.inc
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
new file mode 100644
index 00000000..177a8a6d
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -0,0 +1,36 @@
+# Common part for build chroot filesystem.
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2021 ilbers GmbH
+
+LICENSE = "gpl-2.0"
+LIC_FILES_CHKSUM = "file://${LAYERDIR_core}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
+
+PV = "1.0"
+
+inherit rootfs
+
+SBUILD_CHROOT_PREINSTALL_COMMON = " \
+    fakeroot \
+    build-essential \
+    debhelper \
+"
+
+SBUILD_CHROOT_PREINSTALL ?= " \
+    ${SBUILD_CHROOT_PREINSTALL_COMMON} \
+"
+
+SBUILD_CHROOT_DIR = "${WORKDIR}/rootfs"
+ROOTFSDIR = "${SBUILD_CHROOT_DIR}"
+ROOTFS_PACKAGES = "${SBUILD_CHROOT_PREINSTALL}"
+
+# We don't need /etc/apt/sources.list.d/isar-apt.list' while it's handled by sbuild
+ROOTFS_CONFIGURE_COMMAND_remove = "rootfs_configure_isar_apt"
+
+DEPLOY_SCHROOT = "${@d.getVar('SCHROOT_' + d.getVar('SBUILD_VARIANT').upper() + '_DIR')}"
+
+do_sbuildchroot_deploy[dirs] = "${DEPLOY_DIR}/schroot-${SBUILD_VARIANT}"
+do_sbuildchroot_deploy() {
+    ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_SCHROOT}"
+}
+addtask sbuildchroot_deploy before do_build after do_rootfs
-- 
2.20.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* [PATCH v3 11/20] sbuild: Introduce a class for another build method
  2021-12-08 13:05 [PATCH v3 00/20] Sbuild/Schroot migration Uladzimir Bely
                   ` (9 preceding siblings ...)
  2021-12-08 13:05 ` [PATCH v3 10/20] sbuild: Add recipes for host and target rootfs to run sbuild Uladzimir Bely
@ 2021-12-08 13:05 ` Uladzimir Bely
  2021-12-08 13:05 ` [PATCH v3 12/20] dpkg: Build packages with sbuild Uladzimir Bely
                   ` (9 subsequent siblings)
  20 siblings, 0 replies; 23+ messages in thread
From: Uladzimir Bely @ 2021-12-08 13:05 UTC (permalink / raw)
  To: isar-users

This also adds mounts for base-apt inside schroot and adds import/export
deb files to/from schroot. So that it becomes possible to run second
 `cached` build from local base-apt repo.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/sbuild.bbclass | 146 ++++++++++++++++++++++++++++++++++++
 meta/conf/bitbake.conf      |   2 +
 2 files changed, 148 insertions(+)
 create mode 100644 meta/classes/sbuild.bbclass

diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
new file mode 100644
index 00000000..208b00e1
--- /dev/null
+++ b/meta/classes/sbuild.bbclass
@@ -0,0 +1,146 @@
+# This software is a part of ISAR.
+# Copyright (C) 2021 ilbers GmbH
+
+SCHROOT_CONF ?= "/etc/schroot"
+
+SCHROOT_MOUNTS ?= ""
+
+python __anonymous() {
+    import pwd
+    d.setVar('SCHROOT_USER', pwd.getpwuid(os.geteuid()).pw_name)
+    d.setVar('SCHROOT_USER_HOME', pwd.getpwuid(os.geteuid()).pw_dir)
+
+    mode = d.getVar('ISAR_CROSS_COMPILE', True)
+    distro_arch = d.getVar('DISTRO_ARCH')
+    if mode == "0" or d.getVar('HOST_ARCH') ==  distro_arch or \
+       (d.getVar('HOST_DISTRO') == "debian-stretch" and distro_arch == "i386"):
+        d.setVar('SBUILD_HOST_ARCH', distro_arch)
+        d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_TARGET_DIR'))
+        dep = "sbuild-chroot-target:do_build"
+    else:
+        d.setVar('SBUILD_HOST_ARCH', d.getVar('HOST_ARCH'))
+        d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_HOST_DIR'))
+        dep = "sbuild-chroot-host:do_build"
+    d.setVar('SCHROOT_DEP', dep)
+}
+
+SBUILD_CHROOT ?= "${DEBDISTRONAME}-${SCHROOT_USER}-${@os.getpid()}"
+SBUILD_CHROOT_RW ?= "${SBUILD_CHROOT}-rw"
+
+SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}"
+SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}"
+SCHROOT_CONF_FILE_RW ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}-rw"
+
+schroot_create_configs() {
+    sudo -s <<'EOSUDO'
+        set -e
+
+        cat << EOF > "${SCHROOT_CONF_FILE}"
+[${SBUILD_CHROOT}]
+type=directory
+directory=${SCHROOT_DIR}
+profile=${SBUILD_CHROOT}
+users=${SCHROOT_USER}
+groups=root,sbuild
+root-users=${SCHROOT_USER}
+root-groups=root,sbuild
+source-root-users=${SCHROOT_USER}
+source-root-groups=root,sbuild
+union-type=overlay
+preserve-environment=true
+EOF
+
+        cat << EOF > "${SCHROOT_CONF_FILE_RW}"
+[${SBUILD_CHROOT_RW}]
+type=directory
+directory=${SCHROOT_DIR}
+profile=${SBUILD_CHROOT}
+users=${SCHROOT_USER}
+groups=root,sbuild
+root-users=${SCHROOT_USER}
+root-groups=root,sbuild
+preserve-environment=true
+EOF
+
+        mkdir -p "${SCHROOT_DIR}/etc/apt/preferences.d"
+        cat << EOF > "${SCHROOT_DIR}/etc/apt/preferences.d/isar-apt"
+Package: *
+Pin: release n=${DEBDISTRONAME}
+Pin-Priority: 1000
+EOF
+
+        # Prepare mount points
+        cp -rf "${SCHROOT_CONF}/sbuild" "${SBUILD_CONF_DIR}"
+        sbuild_fstab="${SBUILD_CONF_DIR}/fstab"
+
+        fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind 0 0"
+        grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}
+
+        if [ -d ${DL_DIR} ]; then
+            fstab_downloads="${DL_DIR} /downloads none rw,bind 0 0"
+            grep -qxF "${fstab_downloads}" ${sbuild_fstab} || echo "${fstab_downloads}" >> ${sbuild_fstab}
+        fi
+EOSUDO
+}
+
+schroot_delete_configs() {
+    sudo -s <<'EOSUDO'
+        set -e
+        if [ -d "${SBUILD_CONF_DIR}" ]; then
+            rm -rf "${SBUILD_CONF_DIR}"
+        fi
+        rm -f "${SCHROOT_DIR}/etc/apt/preferences.d/isar-apt"
+        rm -f "${SCHROOT_CONF_FILE}"
+        rm -f "${SCHROOT_CONF_FILE_RW}"
+EOSUDO
+}
+
+schroot_install() {
+    schroot_create_configs
+    APTS="$1"
+
+    distro="${DISTRO}"
+    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+       distro="${HOST_DISTRO}"
+    fi
+
+    deb_dl_dir_import ${SCHROOT_DIR} ${distro}
+    schroot -d / -c ${SBUILD_CHROOT_RW} -u root -- \
+        apt install -y -o Debug::pkgProblemResolver=yes \
+                    --no-install-recommends --download-only ${APTS}
+    deb_dl_dir_export ${SCHROOT_DIR} ${distro}
+
+    schroot -d / -c ${SBUILD_CHROOT_RW} -u root -- \
+        apt install -y -o Debug::pkgProblemResolver=yes \
+                    --no-install-recommends ${APTS}
+    schroot_delete_configs
+}
+
+insert_mounts() {
+    sudo -s <<'EOSUDO'
+        set -e
+        for mp in ${SCHROOT_MOUNTS}; do
+            FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
+            grep -qxF "${FSTAB_LINE}" ${SBUILD_CONF_DIR}/fstab || \
+                echo "${FSTAB_LINE}" >> ${SBUILD_CONF_DIR}/fstab
+        done
+EOSUDO
+}
+
+remove_mounts() {
+    sudo -s <<'EOSUDO'
+        set -e
+        for mp in ${SCHROOT_MOUNTS}; do
+            FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
+            sed -i "\|${FSTAB_LINE}|d" ${SBUILD_CONF_DIR}/fstab
+        done
+EOSUDO
+}
+
+schroot_run() {
+    schroot_create_configs
+    insert_mounts
+    schroot $@
+    remove_mounts
+    schroot_delete_configs
+}
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index cd1c4a64..c1458f05 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -58,6 +58,8 @@ SSTATE_DIR ?= "${TOPDIR}/sstate-cache"
 SSTATE_MANIFESTS = "${TMPDIR}/sstate-control/${DISTRO}-${DISTRO_ARCH}"
 BUILDCHROOT_HOST_DIR = "${DEPLOY_DIR_BUILDCHROOT}-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
 BUILDCHROOT_TARGET_DIR = "${DEPLOY_DIR_BUILDCHROOT}-target/${DISTRO}-${DISTRO_ARCH}"
+SCHROOT_HOST_DIR = "${DEPLOY_DIR}/schroot-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
+SCHROOT_TARGET_DIR = "${DEPLOY_DIR}/schroot-target/${DISTRO}-${DISTRO_ARCH}"
 SDKCHROOT_DIR = "${DEPLOY_DIR_SDKCHROOT}/${DISTRO}-${DISTRO_ARCH}"
 CACHE = "${TMPDIR}/cache"
 KERNEL_FILE ?= "vmlinuz"
-- 
2.20.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* [PATCH v3 12/20] dpkg: Build packages with sbuild
  2021-12-08 13:05 [PATCH v3 00/20] Sbuild/Schroot migration Uladzimir Bely
                   ` (10 preceding siblings ...)
  2021-12-08 13:05 ` [PATCH v3 11/20] sbuild: Introduce a class for another build method Uladzimir Bely
@ 2021-12-08 13:05 ` Uladzimir Bely
  2021-12-08 13:05 ` [PATCH v3 13/20] sbuild: Introduce environment variables export API Uladzimir Bely
                   ` (8 subsequent siblings)
  20 siblings, 0 replies; 23+ messages in thread
From: Uladzimir Bely @ 2021-12-08 13:05 UTC (permalink / raw)
  To: isar-users

From: Anton Mikanovich <amikan@ilbers.de>

Use previously exported to schroot rootfs to build the package with
sbuild tool. Provide isar-apt as extra repository to be automatically
add to apt sources.

Also added /home/.git-downloads mount while it's used
as git alternates location.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 13 +++++++++++++
 meta/classes/dpkg-gbp.bbclass  |  2 ++
 meta/classes/dpkg.bbclass      | 14 ++++++++++++--
 3 files changed, 27 insertions(+), 2 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 6704385b..18c80411 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -4,6 +4,7 @@
 #
 # SPDX-License-Identifier: MIT
 
+inherit sbuild
 inherit buildchroot
 inherit debianize
 inherit terminal
@@ -14,6 +15,8 @@ DEPENDS ?= ""
 
 DEPENDS_append_riscv64 = "${@' crossbuild-essential-riscv64' if d.getVar('ISAR_CROSS_COMPILE', True) == '1' and d.getVar('PN') != 'crossbuild-essential-riscv64' else ''}"
 
+ISAR_APT_REPO ?= "deb [trusted=yes] file:///home/builder/${PN}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO} ${DEBDISTRONAME} main"
+
 python do_adjust_git() {
     import subprocess
 
@@ -196,6 +199,14 @@ dpkg_undo_mounts() {
     sudo rmdir ${BUILDROOT}
 }
 
+do_prepare_build_append() {
+    # Make a local copy of isar-apt repo that is not affected by other parallel builds
+    mkdir -p ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
+    rm -rf ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/*
+    cp -Rl ${REPO_ISAR_DIR} ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
+}
+do_prepare_build[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
+
 # Placeholder for actual dpkg_runbuild() implementation
 dpkg_runbuild() {
     die "This should never be called, overwrite it in your derived class"
@@ -239,6 +250,8 @@ python do_dpkg_build_setscene() {
 addtask dpkg_build_setscene
 do_dpkg_build_setscene[dirs] += "${S}/.."
 
+do_dpkg_build[depends] = "${SCHROOT_DEP}"
+
 KEEP_INSTALLED_ON_CLEAN ?= "0"
 
 CLEANFUNCS += "deb_clean"
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index 2fc37b2f..7eda9b03 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -11,6 +11,8 @@ PATCHTOOL ?= "git"
 
 GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
 
+SCHROOT_MOUNTS = "${WORKDIR}:${PP} ${GITDIR}:/home/.git-downloads"
+
 dpkg_runbuild_prepend() {
     sh -c "
         cd ${WORKDIR}/${PPS}
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 7da73341..6cd0f7d1 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -29,10 +29,20 @@ do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
 
 addtask devshell after do_install_builddeps
 
+
 # Build package from sources using build script
 dpkg_runbuild() {
     E="${@ isar_export_proxies(d)}"
     export PARALLEL_MAKE="${PARALLEL_MAKE}"
-    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
-         /isar/build.sh ${PP}/${PPS} ${PACKAGE_ARCH} ${USE_CCACHE}
+
+    schroot_create_configs
+
+    sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
+        --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \
+        --starting-build-commands="runuser -u ${SCHROOT_USER} -- sh -c \"${SBUILD_PREBUILD:-:}\"" \
+        --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
+        --debbuildopts="-d --source-option=-I" \
+        --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
+
+    schroot_delete_configs
 }
-- 
2.20.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* [PATCH v3 13/20] sbuild: Introduce environment variables export API
  2021-12-08 13:05 [PATCH v3 00/20] Sbuild/Schroot migration Uladzimir Bely
                   ` (11 preceding siblings ...)
  2021-12-08 13:05 ` [PATCH v3 12/20] dpkg: Build packages with sbuild Uladzimir Bely
@ 2021-12-08 13:05 ` Uladzimir Bely
  2021-12-08 13:05 ` [PATCH v3 14/20] linux-custom: Prepare kernel config inside sbuild Uladzimir Bely
                   ` (7 subsequent siblings)
  20 siblings, 0 replies; 23+ messages in thread
From: Uladzimir Bely @ 2021-12-08 13:05 UTC (permalink / raw)
  To: isar-users

From: Anton Mikanovich <amikan@ilbers.de>

In case there is no ability to pass variables through template
mechanism we need to keep fallback API to migrate from bare export.
Using it with optee-os-stm32mp15x recipe as example.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 .../optee-os/optee-os-stm32mp15x_3.11.0.bb          |  2 +-
 meta/classes/sbuild.bbclass                         | 13 +++++++++++++
 2 files changed, 14 insertions(+), 1 deletion(-)

diff --git a/meta-isar/recipes-bsp/optee-os/optee-os-stm32mp15x_3.11.0.bb b/meta-isar/recipes-bsp/optee-os/optee-os-stm32mp15x_3.11.0.bb
index 0fa2005d..19aa763a 100644
--- a/meta-isar/recipes-bsp/optee-os/optee-os-stm32mp15x_3.11.0.bb
+++ b/meta-isar/recipes-bsp/optee-os/optee-os-stm32mp15x_3.11.0.bb
@@ -21,5 +21,5 @@ OPTEE_BINARIES = "tee-header_v2.stm32 tee-pageable_v2.stm32 tee-pager_v2.stm32"
 # Set version manually to PV, the tarball does not contain any hint.
 # Alternative: pull from git and add git as build dependency.
 dpkg_runbuild_prepend() {
-    export TEE_IMPL_VERSION=${PV}
+    sbuild_export TEE_IMPL_VERSION=${PV}
 }
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 208b00e1..3471595a 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -95,6 +95,19 @@ schroot_delete_configs() {
 EOSUDO
 }
 
+sbuild_export() {
+    SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
+    VAR_LINE="'${1%%=*}' => '${1#*=}',"
+    if [ -s "${SBUILD_CONFIG}" ]; then
+        sed -i -e "\$i\\" -e "${VAR_LINE}" ${SBUILD_CONFIG}
+    else
+        echo "\$build_environment = {" > ${SBUILD_CONFIG}
+        echo "${VAR_LINE}" >> ${SBUILD_CONFIG}
+        echo "};" >> ${SBUILD_CONFIG}
+    fi
+    export SBUILD_CONFIG="${SBUILD_CONFIG}"
+}
+
 schroot_install() {
     schroot_create_configs
     APTS="$1"
-- 
2.20.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* [PATCH v3 14/20] linux-custom: Prepare kernel config inside sbuild
  2021-12-08 13:05 [PATCH v3 00/20] Sbuild/Schroot migration Uladzimir Bely
                   ` (12 preceding siblings ...)
  2021-12-08 13:05 ` [PATCH v3 13/20] sbuild: Introduce environment variables export API Uladzimir Bely
@ 2021-12-08 13:05 ` Uladzimir Bely
  2021-12-08 13:05 ` [PATCH v3 15/20] dpkg: Remove builddeps install task Uladzimir Bely
                   ` (6 subsequent siblings)
  20 siblings, 0 replies; 23+ messages in thread
From: Uladzimir Bely @ 2021-12-08 13:05 UTC (permalink / raw)
  To: isar-users

From: Anton Mikanovich <amikan@ilbers.de>

Move prepare code inside sbuild hook to perform all the operations
inside schroot.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/recipes-kernel/linux/linux-custom.inc | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index 57740860..92c5e4a7 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -161,9 +161,9 @@ dpkg_configure_kernel() {
 		(cd ${WORKDIR} && cp ${src_frags} ${S}/debian/fragments/)
 	fi
 
-	sudo -E chroot --userspec=$(id -u):$(id -g) ${BUILDCHROOT_DIR} sh -c " \
+	SBUILD_PREBUILD="                                                      \
 		export ARCH=${KERNEL_ARCH} &&                                  \
-		cd ${PP}/${PPS} &&                                             \
+		cd %p &&                                                       \
 		make O=${KERNEL_BUILD_DIR} ${config_target} &&                 \
 		./scripts/kconfig/merge_config.sh                              \
 			-O ${KERNEL_BUILD_DIR}/                                \
-- 
2.20.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* [PATCH v3 15/20] dpkg: Remove builddeps install task.
  2021-12-08 13:05 [PATCH v3 00/20] Sbuild/Schroot migration Uladzimir Bely
                   ` (13 preceding siblings ...)
  2021-12-08 13:05 ` [PATCH v3 14/20] linux-custom: Prepare kernel config inside sbuild Uladzimir Bely
@ 2021-12-08 13:05 ` Uladzimir Bely
  2021-12-08 13:05 ` [PATCH v3 16/20] dpkg-base: Switch devshell to use schroot Uladzimir Bely
                   ` (5 subsequent siblings)
  20 siblings, 0 replies; 23+ messages in thread
From: Uladzimir Bely @ 2021-12-08 13:05 UTC (permalink / raw)
  To: isar-users

While builddeps are now handled interlnally by sbuild, we don't need
a task to preinstall them in buildchroot.

But for the second local (base-apt) build we need a way to keep
all dependencies in form of .deb files in DL_DIR. This is done
by executing additional commands in sbuild that copy them to/from
an externally mounted folder.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass | 43 ++++++++++++++++++---------------------
 1 file changed, 20 insertions(+), 23 deletions(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 6cd0f7d1..fe5a6011 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -5,30 +5,7 @@ inherit dpkg-base
 
 PACKAGE_ARCH ?= "${DISTRO_ARCH}"
 
-# Install build dependencies for package
-do_install_builddeps() {
-    dpkg_do_mounts
-    E="${@ isar_export_proxies(d)}"
-    distro="${DISTRO}"
-    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
-       distro="${HOST_DISTRO}"
-    fi
-    deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
-        ${PP}/${PPS} ${PACKAGE_ARCH} --download-only
-    deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
-    sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
-        ${PP}/${PPS} ${PACKAGE_ARCH}
-    dpkg_undo_mounts
-}
-
-addtask install_builddeps after do_prepare_build before do_dpkg_build
 do_install_builddeps[depends] += "isar-apt:do_cache_config"
-# apt and reprepro may not run in parallel, acquire the Isar lock
-do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
-
-addtask devshell after do_install_builddeps
-
 
 # Build package from sources using build script
 dpkg_runbuild() {
@@ -37,12 +14,32 @@ dpkg_runbuild() {
 
     schroot_create_configs
 
+    distro="${DISTRO}"
+    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+        distro="${HOST_DISTRO}"
+    fi
+
+    deb_dl_dir_import "${WORKDIR}/rootfs" "${distro}"
+
+    deb_dir="/var/cache/apt/archives/"
+    ext_deb_dir="/home/builder/${PN}/rootfs/${deb_dir}"
+
+    ( flock 9
+        grep -qxF '$apt_keep_downloaded_packages = 1;' ${SCHROOT_USER_HOME}/.sbuildrc ||
+            echo '$apt_keep_downloaded_packages = 1;' >> ${SCHROOT_USER_HOME}/.sbuildrc
+    ) 9>"${TMPDIR}/sbuildrc.lock"
+
     sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
         --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \
         --starting-build-commands="runuser -u ${SCHROOT_USER} -- sh -c \"${SBUILD_PREBUILD:-:}\"" \
         --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
+        --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
+        --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
+        --finished-build-commands="cp -n --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
         --debbuildopts="-d --source-option=-I" \
         --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
 
+    deb_dl_dir_export "${WORKDIR}/rootfs" "${distro}"
+
     schroot_delete_configs
 }
-- 
2.20.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* [PATCH v3 16/20] dpkg-base: Switch devshell to use schroot
  2021-12-08 13:05 [PATCH v3 00/20] Sbuild/Schroot migration Uladzimir Bely
                   ` (14 preceding siblings ...)
  2021-12-08 13:05 ` [PATCH v3 15/20] dpkg: Remove builddeps install task Uladzimir Bely
@ 2021-12-08 13:05 ` Uladzimir Bely
  2021-12-08 13:05 ` [PATCH v3 17/20] dpkg-base: Switch apt_fetch and apt_unpack " Uladzimir Bely
                   ` (4 subsequent siblings)
  20 siblings, 0 replies; 23+ messages in thread
From: Uladzimir Bely @ 2021-12-08 13:05 UTC (permalink / raw)
  To: isar-users

While packages are now build in schroot with sbuild, devshell
task should make terminal open inside sbuild environment.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-base.bbclass                | 22 ++++++++++++++-----
 meta/classes/sbuild.bbclass                   |  3 +++
 .../sbuild-chroot/sbuild-chroot.inc           |  2 ++
 3 files changed, 22 insertions(+), 5 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 18c80411..34f6e7ab 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -288,16 +288,28 @@ do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
 do_deploy_deb[dirs] = "${S}"
 
 python do_devshell() {
-    bb.build.exec_func('dpkg_do_mounts', d)
+    bb.build.exec_func('schroot_create_configs', d)
 
     isar_export_proxies(d)
 
-    buildchroot = d.getVar('BUILDCHROOT_DIR')
+    schroot = d.getVar('SBUILD_CHROOT')
+    isar_apt = d.getVar('ISAR_APT_REPO')
     pp_pps = os.path.join(d.getVar('PP'), d.getVar('PPS'))
-    termcmd = "sudo -E chroot {0} sh -c 'cd {1}; $SHELL -i'"
-    oe_terminal(termcmd.format(buildchroot, pp_pps), "Isar devshell", d)
 
-    bb.build.exec_func('dpkg_undo_mounts', d)
+    install_deps = ":" if d.getVar('BB_CURRENTTASK') == "devshell_nodeps" else "mk-build-deps -i -t \
+        \"apt-get -y -q -o Debug::pkgProblemResolver=yes --no-install-recommends --allow-downgrades\" \
+        debian/control"
+
+    termcmd = "schroot -d / -c {0} -u root -- sh -c ' \
+        cd {1}; \
+        echo {2} > /etc/apt/sources.list.d/isar_apt.list; \
+        apt-get -y -q update; \
+        {3}; \
+        $SHELL -i \
+    '"
+    oe_terminal(termcmd.format(schroot, pp_pps, isar_apt, install_deps), "Isar devshell", d)
+
+    bb.build.exec_func('schroot_delete_configs', d)
 }
 
 addtask devshell after do_prepare_build
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 3471595a..0df8cb90 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -76,6 +76,9 @@ EOF
         fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind 0 0"
         grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}
 
+        fstab_pkgdir="${WORKDIR} /home/builder/${PN} none rw,bind 0 0"
+        grep -qxF "${fstab_pkgdir}" ${sbuild_fstab} || echo "${fstab_pkgdir}" >> ${sbuild_fstab}
+
         if [ -d ${DL_DIR} ]; then
             fstab_downloads="${DL_DIR} /downloads none rw,bind 0 0"
             grep -qxF "${fstab_downloads}" ${sbuild_fstab} || echo "${fstab_downloads}" >> ${sbuild_fstab}
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
index 177a8a6d..7b41d11d 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -14,6 +14,8 @@ SBUILD_CHROOT_PREINSTALL_COMMON = " \
     fakeroot \
     build-essential \
     debhelper \
+    devscripts \
+    equivs \
 "
 
 SBUILD_CHROOT_PREINSTALL ?= " \
-- 
2.20.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* [PATCH v3 17/20] dpkg-base: Switch apt_fetch and apt_unpack to use schroot
  2021-12-08 13:05 [PATCH v3 00/20] Sbuild/Schroot migration Uladzimir Bely
                   ` (15 preceding siblings ...)
  2021-12-08 13:05 ` [PATCH v3 16/20] dpkg-base: Switch devshell to use schroot Uladzimir Bely
@ 2021-12-08 13:05 ` Uladzimir Bely
  2021-12-08 13:05 ` [PATCH v3 18/20] dpkg-base: Cleanup from buildchroot parts Uladzimir Bely
                   ` (3 subsequent siblings)
  20 siblings, 0 replies; 23+ messages in thread
From: Uladzimir Bely @ 2021-12-08 13:05 UTC (permalink / raw)
  To: isar-users

This moves downloading and unpacking deb-src to schroot environment.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 26 ++++++++------------------
 1 file changed, 8 insertions(+), 18 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 34f6e7ab..f10f424f 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -99,34 +99,25 @@ python() {
 }
 
 do_apt_fetch() {
-    dpkg_do_mounts
-    E="${@ isar_export_proxies(d)}"
-    sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update \
-        -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
-        -o Dir::Etc::SourceParts="-" \
-        -o APT::Get::List-Cleanup="0"
-
+    schroot_create_configs
     for uri in "${SRC_APT}"; do
-        sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+        schroot -d / -c ${SBUILD_CHROOT} -- \
             sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only-source source "$2"' my_script "${DISTRO}" "${uri}"
     done
-
-    dpkg_undo_mounts
+    schroot_delete_configs
 }
 
 addtask apt_fetch after do_unpack before do_apt_unpack
 do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
 
-# Add dependency from the correct buildchroot: host or target
-do_apt_fetch[depends] = "${BUILDCHROOT_DEP}"
+# Add dependency from the correct schroot: host or target
+do_apt_fetch[depends] = "${SCHROOT_DEP}"
 
 do_apt_unpack() {
     rm -rf ${S}
-    dpkg_do_mounts
-    E="${@ isar_export_proxies(d)}"
-
+    schroot_create_configs
     for uri in "${SRC_APT}"; do
-        sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+        schroot -d / -c ${SBUILD_CHROOT} -- \
             sh -c ' \
                 set -e
                 dscfile="$(apt-get -y -qq --print-uris --only-source source "${2}" | cut -d " " -f2 | grep -E "*.dsc")"
@@ -135,8 +126,7 @@ do_apt_unpack() {
                 dpkg-source -x "${dscfile}" "${PPS}"' \
                     my_script "${DISTRO}" "${uri}"
     done
-
-    dpkg_undo_mounts
+    schroot_delete_configs
 }
 
 addtask apt_unpack after do_apt_fetch before do_patch
-- 
2.20.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* [PATCH v3 18/20] dpkg-base: Cleanup from buildchroot parts.
  2021-12-08 13:05 [PATCH v3 00/20] Sbuild/Schroot migration Uladzimir Bely
                   ` (16 preceding siblings ...)
  2021-12-08 13:05 ` [PATCH v3 17/20] dpkg-base: Switch apt_fetch and apt_unpack " Uladzimir Bely
@ 2021-12-08 13:05 ` Uladzimir Bely
  2021-12-08 13:05 ` [PATCH v3 19/20] doc: Add sbuild-related documentation Uladzimir Bely
                   ` (2 subsequent siblings)
  20 siblings, 0 replies; 23+ messages in thread
From: Uladzimir Bely @ 2021-12-08 13:05 UTC (permalink / raw)
  To: isar-users

This removes remaining buildchroot-related tasks and variables
from dpkg-base class while there are not used anymore.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 43 +++-------------------------------
 meta/classes/dpkg.bbclass      |  4 ----
 2 files changed, 3 insertions(+), 44 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index f10f424f..e7acace6 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -5,7 +5,6 @@
 # SPDX-License-Identifier: MIT
 
 inherit sbuild
-inherit buildchroot
 inherit debianize
 inherit terminal
 inherit repository
@@ -165,29 +164,7 @@ addtask prepare_build after do_patch do_transform_template before do_dpkg_build
 # deployed to isar-apt
 do_prepare_build[deptask] = "do_deploy_deb"
 
-BUILDROOT = "${BUILDCHROOT_DIR}/${PP}"
-
-dpkg_do_mounts() {
-    mkdir -p ${BUILDROOT}
-    sudo mount --bind ${WORKDIR} ${BUILDROOT}
-
-    buildchroot_do_mounts
-}
-
-dpkg_undo_mounts() {
-    i=0
-    while ! sudo umount ${BUILDROOT}; do
-        sleep 0.1
-        if [ `expr $i % 100` -eq 0 ] ; then
-            bbwarn "${BUILDROOT}: Couldn't unmount ($i), retrying..."
-        fi
-        if [ $i -ge 10000 ]; then
-            bbfatal "${BUILDROOT}: Couldn't unmount after timeout"
-        fi
-        i=`expr $i + 1`
-    done
-    sudo rmdir ${BUILDROOT}
-}
+do_prepare_build[depends] = "${SCHROOT_DEP}"
 
 do_prepare_build_append() {
     # Make a local copy of isar-apt repo that is not affected by other parallel builds
@@ -203,14 +180,11 @@ dpkg_runbuild() {
 }
 
 python do_dpkg_build() {
-    lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
-                             shared=True)
-    bb.build.exec_func("dpkg_do_mounts", d)
+    bb.build.exec_func('schroot_create_configs', d)
     try:
         bb.build.exec_func("dpkg_runbuild", d)
     finally:
-        bb.build.exec_func("dpkg_undo_mounts", d)
-        bb.utils.unlockfile(lock)
+        bb.build.exec_func('schroot_delete_configs', d)
 }
 
 addtask dpkg_build
@@ -242,8 +216,6 @@ do_dpkg_build_setscene[dirs] += "${S}/.."
 
 do_dpkg_build[depends] = "${SCHROOT_DEP}"
 
-KEEP_INSTALLED_ON_CLEAN ?= "0"
-
 CLEANFUNCS += "deb_clean"
 
 deb_clean() {
@@ -252,15 +224,6 @@ deb_clean() {
         for d in ${DEBS}; do
             repo_del_package "${REPO_ISAR_DIR}"/"${DISTRO}" \
                 "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${d}"
-            if [ "${KEEP_INSTALLED_ON_CLEAN}" = "1" ]; then
-                continue;
-            fi
-            package=$(basename "${d}")
-            package_remove="/usr/bin/apt-get remove -y ${package%%_*}"
-            sudo -E chroot ${BUILDCHROOT_DIR} ${package_remove} || true
-            if [ "${BUILDCHROOT_DIR}" != "${BUILDCHROOT_TARGET_DIR}" ]; then
-                    sudo -E chroot ${BUILDCHROOT_TARGET_DIR} ${package_remove} || true
-            fi
         done
     fi
 }
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index fe5a6011..85bb1300 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -12,8 +12,6 @@ dpkg_runbuild() {
     E="${@ isar_export_proxies(d)}"
     export PARALLEL_MAKE="${PARALLEL_MAKE}"
 
-    schroot_create_configs
-
     distro="${DISTRO}"
     if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
         distro="${HOST_DISTRO}"
@@ -40,6 +38,4 @@ dpkg_runbuild() {
         --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
 
     deb_dl_dir_export "${WORKDIR}/rootfs" "${distro}"
-
-    schroot_delete_configs
 }
-- 
2.20.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* [PATCH v3 19/20] doc: Add sbuild-related documentation
  2021-12-08 13:05 [PATCH v3 00/20] Sbuild/Schroot migration Uladzimir Bely
                   ` (17 preceding siblings ...)
  2021-12-08 13:05 ` [PATCH v3 18/20] dpkg-base: Cleanup from buildchroot parts Uladzimir Bely
@ 2021-12-08 13:05 ` Uladzimir Bely
  2021-12-08 13:05 ` [PATCH v3 20/20] sbuild: add ccache support Uladzimir Bely
  2021-12-08 13:40 ` [PATCH v3 00/20] Sbuild/Schroot migration Uladzimir Bely
  20 siblings, 0 replies; 23+ messages in thread
From: Uladzimir Bely @ 2021-12-08 13:05 UTC (permalink / raw)
  To: isar-users

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 doc/user_manual.md | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 9017dd89..088cbe18 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -84,6 +84,24 @@ apt install \
   pristine-tar
 ```
 
+Additional setup is required since `sbuild` is now used for package build.
+Install the following packages:
+```
+apt install \
+  sbuild \
+  schroot
+```
+Also, user who runs isar should be added to `sbuild` group.
+
+**NOTE:** sbuild version (<=0.78.1) packaged in Debian Buster doesn't support
+`$apt_keep_downloaded_packages` option which is required in Isar for
+populating `${DL_DIR}/deb`. So, host `sbuild` in this case should be manually
+upgraded to >=0.81.2 version from Debian Bullseye.
+
+```
+sudo gpasswd -a <username> sbuild
+```
+
 If your host is >= buster, also install the following package.
 ```
 apt install python3-distutils
-- 
2.20.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* [PATCH v3 20/20] sbuild: add ccache support
  2021-12-08 13:05 [PATCH v3 00/20] Sbuild/Schroot migration Uladzimir Bely
                   ` (18 preceding siblings ...)
  2021-12-08 13:05 ` [PATCH v3 19/20] doc: Add sbuild-related documentation Uladzimir Bely
@ 2021-12-08 13:05 ` Uladzimir Bely
  2021-12-08 13:40 ` [PATCH v3 00/20] Sbuild/Schroot migration Uladzimir Bely
  20 siblings, 0 replies; 23+ messages in thread
From: Uladzimir Bely @ 2021-12-08 13:05 UTC (permalink / raw)
  To: isar-users

This adds ccache support for custom packages in the same manner
as it was done previously for buildchroot-based build, by using
USE_CCACHE variable in local.conf or per-recipe.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass                     |  4 ++++
 meta/classes/sbuild.bbclass                   | 22 +++++++++++++++++++
 .../sbuild-chroot/sbuild-chroot.inc           |  1 +
 3 files changed, 27 insertions(+)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 85bb1300..e68605ef 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -27,6 +27,10 @@ dpkg_runbuild() {
             echo '$apt_keep_downloaded_packages = 1;' >> ${SCHROOT_USER_HOME}/.sbuildrc
     ) 9>"${TMPDIR}/sbuildrc.lock"
 
+    if [ ${USE_CCACHE} -eq 1 ]; then
+        schroot_configure_ccache
+    fi
+
     sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
         --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \
         --starting-build-commands="runuser -u ${SCHROOT_USER} -- sh -c \"${SBUILD_PREBUILD:-:}\"" \
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 0df8cb90..54ae8550 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -160,3 +160,25 @@ schroot_run() {
     remove_mounts
     schroot_delete_configs
 }
+
+schroot_configure_ccache() {
+    sudo -s <<'EOSUDO'
+        set -e
+
+        sbuild_fstab="${SBUILD_CONF_DIR}/fstab"
+
+        install --group=sbuild --mode=2775 -d ${CCACHE_DIR}
+        fstab_ccachedir="${CCACHE_DIR} /ccache none rw,bind 0 0"
+        grep -qxF "${fstab_ccachedir}" ${sbuild_fstab} || echo "${fstab_ccachedir}" >> ${sbuild_fstab}
+
+        cat << END > ${CCACHE_DIR}/sbuild-setup
+#!/bin/sh
+export CCACHE_DIR=/ccache
+export PATH="/usr/lib/ccache:\$PATH"
+exec "\$@"
+END
+        chmod a+rx ${CCACHE_DIR}/sbuild-setup
+
+        echo "command-prefix=/ccache/sbuild-setup" >> "${SCHROOT_CONF_FILE}"
+EOSUDO
+}
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
index 7b41d11d..4688da1d 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -14,6 +14,7 @@ SBUILD_CHROOT_PREINSTALL_COMMON = " \
     fakeroot \
     build-essential \
     debhelper \
+    ccache \
     devscripts \
     equivs \
 "
-- 
2.20.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [PATCH v3 00/20] Sbuild/Schroot migration
  2021-12-08 13:05 [PATCH v3 00/20] Sbuild/Schroot migration Uladzimir Bely
                   ` (19 preceding siblings ...)
  2021-12-08 13:05 ` [PATCH v3 20/20] sbuild: add ccache support Uladzimir Bely
@ 2021-12-08 13:40 ` Uladzimir Bely
  2021-12-08 17:20   ` Jan Kiszka
  20 siblings, 1 reply; 23+ messages in thread
From: Uladzimir Bely @ 2021-12-08 13:40 UTC (permalink / raw)
  To: isar-users; +Cc: Uladzimir Bely

In mail from среда, 8 декабря 2021 г. 16:05:24 +03 user Uladzimir Bely wrote:
> This is a patchset showing how sbuild/schroot tools can be integrated
> into Isar build system.
> 
> Base schroot image is created with sbuild-chroot-target (or
> sbuild-chroot-host in case of cross-build) recipe. These images are based
> on isar-bootstrap ones, but include some build-related stuff preinstalled.
> 
> To use this changes you need to have sbuild and schroot installed and
> configured. Current user should be added to sbuild group.
> 
> If 'kas-container' is used, it should be modified to support sbuild:
> - /var/lib/schroot/union/overlay should be externally mounted (-v option)
> - export KAS_IMAGE_VERSION=2.6.2-sbuild
> - export KAS_CONTAINER_IMAGE_PATH=ghcr.io/wiselord
> Here I uploaded a docker image that has preinstalled software (sbuild,
> schroot, etc) required to run sbuild. Later, the changes might be
> included in official kas docker images.
> 
> If 'gitlab' is used, the similar changes are required:
> - 'image: ghcr.io/wiselord/kas-isar:2.6.2-sbuild' in .gitlab-ci.yml
> - external dir for schroot overlay should also be specified in
> /etc/gitlab-runner/config.toml:
> volumes = ["/path/to/overlay:/var/lib/schroot/union/overlay"]
> 
> The patchset was tested with 'meta-iot2050' and 'xenomai-images'
> downstreams. Some patches are required for these downstreams,
> appropriate pull requests will be created soon.
> 
> Changes since v2:
> - patches reworked/squashed for easier reading and understanding
> - fixed building foreigh architectures with kas-docker
> - implemented support of ccache
> - fixed devshell and devshell_nodeps
> Changes since v1:
> - parallel builds with different BUILD_DIR are supported
> - parallel multiconfig targets in one build are supported
> - per-task schroot configuration in /etc/schroot/ is now used
> - patchset now passes Jenkins CI (so patches changes RFC => PATCH)
> 
> Current limitations:
> - parallel building of several packages requires more free space in
> comparison with buildchroot-based architecture. This happens due
> the sbuild architecture which uses some basic layer (common for all
> packages) and per-package separate layer (where builddeps are downloaded
> and installed).
> 
> For experiments, the code can be taken from Isar 'ubely/sbuild' branch.
> This is a dev branch, so sometimes it can be force-pushed when some
> fixes are applied of rebased to 'next' branch.
> 
> First 9 patches (1..9) - are just preparation patches. With them, Isar will
> continue using buildchroot-based architecture. So, potentially,
> they can be merged as a subseries.
> 
> Remaining 11 patches (10..20) represent a sbuild-related changes.
> 
> Anton Mikanovich (10):
>   template: Make templates passthrough
>   linux-module: Do not use shell environment
>   u-boot: Do not use shell environment
>   trusted-firmware: Do not use shell environment
>   optee-os: Do not use shell environment
>   kselftest: Do not use shell environment
>   linux-mainline: Move cfg fragment test to debian/rules
>   dpkg: Build packages with sbuild
>   sbuild: Introduce environment variables export API
>   linux-custom: Prepare kernel config inside sbuild
> 
> Uladzimir Bely (10):
>   dpkg-gbp: Use separate command to export tarball
>   dpkg-gbp: Use host tools for dsc preparation
>   sbuild: Add recipes for host and target rootfs to run sbuild
>   sbuild: Introduce a class for another build method
>   dpkg: Remove builddeps install task.
>   dpkg-base: Switch devshell to use schroot
>   dpkg-base: Switch apt_fetch and apt_unpack to use schroot
>   dpkg-base: Cleanup from buildchroot parts.
>   doc: Add sbuild-related documentation
>   sbuild: add ccache support
> 
>  doc/user_manual.md                            |  22 ++-
>  .../optee-os/optee-os-stm32mp15x_3.11.0.bb    |   2 +-
>  .../linux/linux-mainline_5.4.70.bb            |  16 +-
>  meta/classes/dpkg-base.bbclass                |  98 ++++------
>  meta/classes/dpkg-gbp.bbclass                 |  26 +--
>  meta/classes/dpkg.bbclass                     |  53 ++---
>  meta/classes/sbuild.bbclass                   | 184 ++++++++++++++++++
>  meta/classes/template.bbclass                 |   7 +-
>  meta/conf/bitbake.conf                        |   2 +
>  .../files/debian/{rules => rules.tmpl}        |   4 +-
>  meta/recipes-bsp/optee-os/optee-os-custom.inc |   9 +-
>  .../files/debian/{rules => rules.tmpl}        |   4 +-
>  .../trusted-firmware-a-custom.inc             |   9 +-
>  .../u-boot/files/debian/{rules => rules.tmpl} |   2 +-
>  meta/recipes-bsp/u-boot/u-boot-custom.inc     |   9 +-
>  .../sbuild-chroot/sbuild-chroot-host.bb       |  13 ++
>  .../sbuild-chroot/sbuild-chroot-target.bb     |  10 +
>  .../sbuild-chroot/sbuild-chroot.inc           |  39 ++++
>  .../kselftest/files/{rules => rules.tmpl}     |   0
>  meta/recipes-kernel/kselftest/kselftest.inc   |  21 +-
>  .../linux-module/files/debian/rules           |  37 ----
>  .../linux-module/files/debian/rules.tmpl      |  52 +++++
>  meta/recipes-kernel/linux-module/module.inc   |  19 +-
>  meta/recipes-kernel/linux/linux-custom.inc    |   7 +-
>  24 files changed, 437 insertions(+), 208 deletions(-)
>  create mode 100644 meta/classes/sbuild.bbclass
>  rename meta/recipes-bsp/optee-os/files/debian/{rules => rules.tmpl} (75%)
>  rename meta/recipes-bsp/trusted-firmware-a/files/debian/{rules =>
> rules.tmpl} (77%) rename meta/recipes-bsp/u-boot/files/debian/{rules =>
> rules.tmpl} (94%) create mode 100644
> meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb create mode
> 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb create
> mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc rename
> meta/recipes-kernel/kselftest/files/{rules => rules.tmpl} (100%) delete
> mode 100755 meta/recipes-kernel/linux-module/files/debian/rules create mode
> 100755 meta/recipes-kernel/linux-module/files/debian/rules.tmpl

Just a question: Does 'xenomai-images' and 'meta-iot2050' have their own 
maillists where I could send the changes to work with isar-based sbuild? I 
just found a maillist for 'kas', but not for these projects.

-- 
Uladzimir Bely
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov



^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [PATCH v3 00/20] Sbuild/Schroot migration
  2021-12-08 13:40 ` [PATCH v3 00/20] Sbuild/Schroot migration Uladzimir Bely
@ 2021-12-08 17:20   ` Jan Kiszka
  0 siblings, 0 replies; 23+ messages in thread
From: Jan Kiszka @ 2021-12-08 17:20 UTC (permalink / raw)
  To: Uladzimir Bely, isar-users

On 08.12.21 14:40, Uladzimir Bely wrote:
> In mail from среда, 8 декабря 2021 г. 16:05:24 +03 user Uladzimir Bely wrote:
>> This is a patchset showing how sbuild/schroot tools can be integrated
>> into Isar build system.
>>
>> Base schroot image is created with sbuild-chroot-target (or
>> sbuild-chroot-host in case of cross-build) recipe. These images are based
>> on isar-bootstrap ones, but include some build-related stuff preinstalled.
>>
>> To use this changes you need to have sbuild and schroot installed and
>> configured. Current user should be added to sbuild group.
>>
>> If 'kas-container' is used, it should be modified to support sbuild:
>> - /var/lib/schroot/union/overlay should be externally mounted (-v option)
>> - export KAS_IMAGE_VERSION=2.6.2-sbuild
>> - export KAS_CONTAINER_IMAGE_PATH=ghcr.io/wiselord
>> Here I uploaded a docker image that has preinstalled software (sbuild,
>> schroot, etc) required to run sbuild. Later, the changes might be
>> included in official kas docker images.
>>
>> If 'gitlab' is used, the similar changes are required:
>> - 'image: ghcr.io/wiselord/kas-isar:2.6.2-sbuild' in .gitlab-ci.yml
>> - external dir for schroot overlay should also be specified in
>> /etc/gitlab-runner/config.toml:
>> volumes = ["/path/to/overlay:/var/lib/schroot/union/overlay"]
>>
>> The patchset was tested with 'meta-iot2050' and 'xenomai-images'
>> downstreams. Some patches are required for these downstreams,
>> appropriate pull requests will be created soon.
>>
>> Changes since v2:
>> - patches reworked/squashed for easier reading and understanding
>> - fixed building foreigh architectures with kas-docker
>> - implemented support of ccache
>> - fixed devshell and devshell_nodeps
>> Changes since v1:
>> - parallel builds with different BUILD_DIR are supported
>> - parallel multiconfig targets in one build are supported
>> - per-task schroot configuration in /etc/schroot/ is now used
>> - patchset now passes Jenkins CI (so patches changes RFC => PATCH)
>>
>> Current limitations:
>> - parallel building of several packages requires more free space in
>> comparison with buildchroot-based architecture. This happens due
>> the sbuild architecture which uses some basic layer (common for all
>> packages) and per-package separate layer (where builddeps are downloaded
>> and installed).
>>
>> For experiments, the code can be taken from Isar 'ubely/sbuild' branch.
>> This is a dev branch, so sometimes it can be force-pushed when some
>> fixes are applied of rebased to 'next' branch.
>>
>> First 9 patches (1..9) - are just preparation patches. With them, Isar will
>> continue using buildchroot-based architecture. So, potentially,
>> they can be merged as a subseries.
>>
>> Remaining 11 patches (10..20) represent a sbuild-related changes.
>>
>> Anton Mikanovich (10):
>>   template: Make templates passthrough
>>   linux-module: Do not use shell environment
>>   u-boot: Do not use shell environment
>>   trusted-firmware: Do not use shell environment
>>   optee-os: Do not use shell environment
>>   kselftest: Do not use shell environment
>>   linux-mainline: Move cfg fragment test to debian/rules
>>   dpkg: Build packages with sbuild
>>   sbuild: Introduce environment variables export API
>>   linux-custom: Prepare kernel config inside sbuild
>>
>> Uladzimir Bely (10):
>>   dpkg-gbp: Use separate command to export tarball
>>   dpkg-gbp: Use host tools for dsc preparation
>>   sbuild: Add recipes for host and target rootfs to run sbuild
>>   sbuild: Introduce a class for another build method
>>   dpkg: Remove builddeps install task.
>>   dpkg-base: Switch devshell to use schroot
>>   dpkg-base: Switch apt_fetch and apt_unpack to use schroot
>>   dpkg-base: Cleanup from buildchroot parts.
>>   doc: Add sbuild-related documentation
>>   sbuild: add ccache support
>>
>>  doc/user_manual.md                            |  22 ++-
>>  .../optee-os/optee-os-stm32mp15x_3.11.0.bb    |   2 +-
>>  .../linux/linux-mainline_5.4.70.bb            |  16 +-
>>  meta/classes/dpkg-base.bbclass                |  98 ++++------
>>  meta/classes/dpkg-gbp.bbclass                 |  26 +--
>>  meta/classes/dpkg.bbclass                     |  53 ++---
>>  meta/classes/sbuild.bbclass                   | 184 ++++++++++++++++++
>>  meta/classes/template.bbclass                 |   7 +-
>>  meta/conf/bitbake.conf                        |   2 +
>>  .../files/debian/{rules => rules.tmpl}        |   4 +-
>>  meta/recipes-bsp/optee-os/optee-os-custom.inc |   9 +-
>>  .../files/debian/{rules => rules.tmpl}        |   4 +-
>>  .../trusted-firmware-a-custom.inc             |   9 +-
>>  .../u-boot/files/debian/{rules => rules.tmpl} |   2 +-
>>  meta/recipes-bsp/u-boot/u-boot-custom.inc     |   9 +-
>>  .../sbuild-chroot/sbuild-chroot-host.bb       |  13 ++
>>  .../sbuild-chroot/sbuild-chroot-target.bb     |  10 +
>>  .../sbuild-chroot/sbuild-chroot.inc           |  39 ++++
>>  .../kselftest/files/{rules => rules.tmpl}     |   0
>>  meta/recipes-kernel/kselftest/kselftest.inc   |  21 +-
>>  .../linux-module/files/debian/rules           |  37 ----
>>  .../linux-module/files/debian/rules.tmpl      |  52 +++++
>>  meta/recipes-kernel/linux-module/module.inc   |  19 +-
>>  meta/recipes-kernel/linux/linux-custom.inc    |   7 +-
>>  24 files changed, 437 insertions(+), 208 deletions(-)
>>  create mode 100644 meta/classes/sbuild.bbclass
>>  rename meta/recipes-bsp/optee-os/files/debian/{rules => rules.tmpl} (75%)
>>  rename meta/recipes-bsp/trusted-firmware-a/files/debian/{rules =>
>> rules.tmpl} (77%) rename meta/recipes-bsp/u-boot/files/debian/{rules =>
>> rules.tmpl} (94%) create mode 100644
>> meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb create mode
>> 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb create
>> mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc rename
>> meta/recipes-kernel/kselftest/files/{rules => rules.tmpl} (100%) delete
>> mode 100755 meta/recipes-kernel/linux-module/files/debian/rules create mode
>> 100755 meta/recipes-kernel/linux-module/files/debian/rules.tmpl
> 
> Just a question: Does 'xenomai-images' and 'meta-iot2050' have their own 
> maillists where I could send the changes to work with isar-based sbuild? I 
> just found a maillist for 'kas', but not for these projects.
> 

For xenomai-images, we handle changes via xenomai@xenomai.org (please
additional tag related patches with "[xenomai-images]").

meta-iot2050 takes pull requests on github.

For both projects, changes that will work with current Isar can already
be submitted. Changes that depend on your pending series could be
proposed along with an Isar revision bump once 'next' picked up your
queue. Or send them earlier with a clear remark that those are for
reference and future use. But it is probably better to wait for the
'next' merge.

Thanks,
Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

^ permalink raw reply	[flat|nested] 23+ messages in thread

end of thread, other threads:[~2021-12-08 17:20 UTC | newest]

Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-12-08 13:05 [PATCH v3 00/20] Sbuild/Schroot migration Uladzimir Bely
2021-12-08 13:05 ` [PATCH v3 01/20] dpkg-gbp: Use separate command to export tarball Uladzimir Bely
2021-12-08 13:05 ` [PATCH v3 02/20] template: Make templates passthrough Uladzimir Bely
2021-12-08 13:05 ` [PATCH v3 03/20] linux-module: Do not use shell environment Uladzimir Bely
2021-12-08 13:05 ` [PATCH v3 04/20] u-boot: " Uladzimir Bely
2021-12-08 13:05 ` [PATCH v3 05/20] trusted-firmware: " Uladzimir Bely
2021-12-08 13:05 ` [PATCH v3 06/20] optee-os: " Uladzimir Bely
2021-12-08 13:05 ` [PATCH v3 07/20] kselftest: " Uladzimir Bely
2021-12-08 13:05 ` [PATCH v3 08/20] linux-mainline: Move cfg fragment test to debian/rules Uladzimir Bely
2021-12-08 13:05 ` [PATCH v3 09/20] dpkg-gbp: Use host tools for dsc preparation Uladzimir Bely
2021-12-08 13:05 ` [PATCH v3 10/20] sbuild: Add recipes for host and target rootfs to run sbuild Uladzimir Bely
2021-12-08 13:05 ` [PATCH v3 11/20] sbuild: Introduce a class for another build method Uladzimir Bely
2021-12-08 13:05 ` [PATCH v3 12/20] dpkg: Build packages with sbuild Uladzimir Bely
2021-12-08 13:05 ` [PATCH v3 13/20] sbuild: Introduce environment variables export API Uladzimir Bely
2021-12-08 13:05 ` [PATCH v3 14/20] linux-custom: Prepare kernel config inside sbuild Uladzimir Bely
2021-12-08 13:05 ` [PATCH v3 15/20] dpkg: Remove builddeps install task Uladzimir Bely
2021-12-08 13:05 ` [PATCH v3 16/20] dpkg-base: Switch devshell to use schroot Uladzimir Bely
2021-12-08 13:05 ` [PATCH v3 17/20] dpkg-base: Switch apt_fetch and apt_unpack " Uladzimir Bely
2021-12-08 13:05 ` [PATCH v3 18/20] dpkg-base: Cleanup from buildchroot parts Uladzimir Bely
2021-12-08 13:05 ` [PATCH v3 19/20] doc: Add sbuild-related documentation Uladzimir Bely
2021-12-08 13:05 ` [PATCH v3 20/20] sbuild: add ccache support Uladzimir Bely
2021-12-08 13:40 ` [PATCH v3 00/20] Sbuild/Schroot migration Uladzimir Bely
2021-12-08 17:20   ` Jan Kiszka

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox