* [PATCH v11 01/21] dpkg-gbp: Use separate command to export tarball
2022-05-23 14:35 [PATCH v11 00/21] Sbuild/Schroot migration Anton Mikanovich
@ 2022-05-23 14:35 ` Anton Mikanovich
2022-05-23 14:35 ` [PATCH v11 02/21] dpkg-gbp: Use host tools for dsc preparation Anton Mikanovich
` (20 subsequent siblings)
21 siblings, 0 replies; 23+ messages in thread
From: Anton Mikanovich @ 2022-05-23 14:35 UTC (permalink / raw)
To: isar-users; +Cc: Uladzimir Bely
From: Uladzimir Bely <ubely@ilbers.de>
We don't actually build the package with gbp, but only prepare it for
the building with external dpkg-buildpackage command. In case there is
no need in real build we can perform only export which will produce
upstream tarball to pass for later building.
This allows to use any build tool for later processing.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
meta/classes/dpkg-gbp.bbclass | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index d956e8c..6129d73 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -30,5 +30,9 @@ do_install_builddeps_append() {
}
dpkg_runbuild_prepend() {
- export GBP_PREFIX="gbp buildpackage --git-ignore-new ${GBP_EXTRA_OPTIONS} --git-builder="
+ sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+ sh -c "cd ${PP}/${PPS} && gbp buildpackage --git-ignore-new --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}"
+ # NOTE: `buildpackage --git-builder=/bin/true --git-pristine-tar` is used
+ # for compatibility with gbp version froms debian-stretch. In newer distros
+ # it's possible to use a subcommand `export-orig --pristine-tar`
}
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH v11 02/21] dpkg-gbp: Use host tools for dsc preparation
2022-05-23 14:35 [PATCH v11 00/21] Sbuild/Schroot migration Anton Mikanovich
2022-05-23 14:35 ` [PATCH v11 01/21] dpkg-gbp: Use separate command to export tarball Anton Mikanovich
@ 2022-05-23 14:35 ` Anton Mikanovich
2022-05-23 14:35 ` [PATCH v11 03/21] sbuild: Add recipes for host and target rootfs to run sbuild Anton Mikanovich
` (19 subsequent siblings)
21 siblings, 0 replies; 23+ messages in thread
From: Anton Mikanovich @ 2022-05-23 14:35 UTC (permalink / raw)
To: isar-users; +Cc: Uladzimir Bely
From: Uladzimir Bely <ubely@ilbers.de>
Instead of preinstalling gbp-related packages in sbuild chroot
we will use their host versions in order to keep sbuild environment
as clean as possible.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
doc/user_manual.md | 4 +++-
meta/classes/dpkg-gbp.bbclass | 24 ++++--------------------
2 files changed, 7 insertions(+), 21 deletions(-)
diff --git a/doc/user_manual.md b/doc/user_manual.md
index cdb7322..bcff8d9 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -81,7 +81,9 @@ apt install \
qemu \
qemu-user-static \
reprepro \
- sudo
+ sudo \
+ git-buildpackage \
+ pristine-tar
```
If your host is >= buster, also install the following package.
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index 6129d73..2b0b818 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -9,29 +9,13 @@ S = "${WORKDIR}/git"
PATCHTOOL ?= "git"
-GBP_DEPENDS ?= "git-buildpackage pristine-tar"
GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
-do_install_builddeps_append() {
- dpkg_do_mounts
- distro="${DISTRO}"
- if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
- distro="${HOST_DISTRO}"
- fi
- deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
- sudo -E chroot ${BUILDCHROOT_DIR} \
- apt-get install -y -o Debug::pkgProblemResolver=yes \
- --no-install-recommends --download-only ${GBP_DEPENDS}
- deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
- sudo -E chroot ${BUILDCHROOT_DIR} \
- apt-get install -y -o Debug::pkgProblemResolver=yes \
- --no-install-recommends ${GBP_DEPENDS}
- dpkg_undo_mounts
-}
-
dpkg_runbuild_prepend() {
- sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
- sh -c "cd ${PP}/${PPS} && gbp buildpackage --git-ignore-new --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}"
+ sh -c "
+ cd ${WORKDIR}/${PPS}
+ gbp buildpackage --git-ignore-new --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}
+ "
# NOTE: `buildpackage --git-builder=/bin/true --git-pristine-tar` is used
# for compatibility with gbp version froms debian-stretch. In newer distros
# it's possible to use a subcommand `export-orig --pristine-tar`
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH v11 03/21] sbuild: Add recipes for host and target rootfs to run sbuild
2022-05-23 14:35 [PATCH v11 00/21] Sbuild/Schroot migration Anton Mikanovich
2022-05-23 14:35 ` [PATCH v11 01/21] dpkg-gbp: Use separate command to export tarball Anton Mikanovich
2022-05-23 14:35 ` [PATCH v11 02/21] dpkg-gbp: Use host tools for dsc preparation Anton Mikanovich
@ 2022-05-23 14:35 ` Anton Mikanovich
2022-05-23 14:36 ` [PATCH v11 04/21] sbuild: Introduce a class for another build method Anton Mikanovich
` (18 subsequent siblings)
21 siblings, 0 replies; 23+ messages in thread
From: Anton Mikanovich @ 2022-05-23 14:35 UTC (permalink / raw)
To: isar-users; +Cc: Uladzimir Bely
From: Uladzimir Bely <ubely@ilbers.de>
Similar to buildchroot, we need a separate rootfs to be used
for schroot.
It's based on bootstrapped rootfs, but includes several
common build-related packages.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
.../sbuild-chroot/sbuild-chroot-host.bb | 13 +++++++
.../sbuild-chroot/sbuild-chroot-target.bb | 10 ++++++
.../sbuild-chroot/sbuild-chroot.inc | 36 +++++++++++++++++++
3 files changed, 59 insertions(+)
create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
new file mode 100644
index 0000000..aa82846
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
@@ -0,0 +1,13 @@
+# Root filesystem for packages building
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2021 ilbers GmbH
+
+DESCRIPTION = "Isar sbuild/schroot filesystem for host"
+
+SBUILD_VARIANT = "host"
+
+require sbuild-chroot.inc
+
+ROOTFS_ARCH = "${HOST_ARCH}"
+ROOTFS_DISTRO = "${HOST_DISTRO}"
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
new file mode 100644
index 0000000..d75d783
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
@@ -0,0 +1,10 @@
+# Root filesystem for packages building
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2021 ilbers GmbH
+
+DESCRIPTION = "Isar sbuild/schroot filesystem for target"
+
+SBUILD_VARIANT = "target"
+
+require sbuild-chroot.inc
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
new file mode 100644
index 0000000..177a8a6
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -0,0 +1,36 @@
+# Common part for build chroot filesystem.
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2021 ilbers GmbH
+
+LICENSE = "gpl-2.0"
+LIC_FILES_CHKSUM = "file://${LAYERDIR_core}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
+
+PV = "1.0"
+
+inherit rootfs
+
+SBUILD_CHROOT_PREINSTALL_COMMON = " \
+ fakeroot \
+ build-essential \
+ debhelper \
+"
+
+SBUILD_CHROOT_PREINSTALL ?= " \
+ ${SBUILD_CHROOT_PREINSTALL_COMMON} \
+"
+
+SBUILD_CHROOT_DIR = "${WORKDIR}/rootfs"
+ROOTFSDIR = "${SBUILD_CHROOT_DIR}"
+ROOTFS_PACKAGES = "${SBUILD_CHROOT_PREINSTALL}"
+
+# We don't need /etc/apt/sources.list.d/isar-apt.list' while it's handled by sbuild
+ROOTFS_CONFIGURE_COMMAND_remove = "rootfs_configure_isar_apt"
+
+DEPLOY_SCHROOT = "${@d.getVar('SCHROOT_' + d.getVar('SBUILD_VARIANT').upper() + '_DIR')}"
+
+do_sbuildchroot_deploy[dirs] = "${DEPLOY_DIR}/schroot-${SBUILD_VARIANT}"
+do_sbuildchroot_deploy() {
+ ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_SCHROOT}"
+}
+addtask sbuildchroot_deploy before do_build after do_rootfs
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH v11 04/21] sbuild: Introduce a class for another build method
2022-05-23 14:35 [PATCH v11 00/21] Sbuild/Schroot migration Anton Mikanovich
` (2 preceding siblings ...)
2022-05-23 14:35 ` [PATCH v11 03/21] sbuild: Add recipes for host and target rootfs to run sbuild Anton Mikanovich
@ 2022-05-23 14:36 ` Anton Mikanovich
2022-05-23 14:36 ` [PATCH v11 05/21] dpkg: Build packages with sbuild Anton Mikanovich
` (17 subsequent siblings)
21 siblings, 0 replies; 23+ messages in thread
From: Anton Mikanovich @ 2022-05-23 14:36 UTC (permalink / raw)
To: isar-users; +Cc: Uladzimir Bely, Anton Mikanovich
From: Uladzimir Bely <ubely@ilbers.de>
This also adds mounts for base-apt inside schroot and adds import/export
deb files to/from schroot. So that it becomes possible to run second
`cached` build from local base-apt repo.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
meta/classes/sbuild.bbclass | 108 ++++++++++++++++++++++++++++++++++++
meta/conf/bitbake.conf | 2 +
2 files changed, 110 insertions(+)
create mode 100644 meta/classes/sbuild.bbclass
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
new file mode 100644
index 0000000..b9fc6a0
--- /dev/null
+++ b/meta/classes/sbuild.bbclass
@@ -0,0 +1,108 @@
+# This software is a part of ISAR.
+# Copyright (C) 2021 ilbers GmbH
+
+SCHROOT_CONF ?= "/etc/schroot"
+
+SCHROOT_MOUNTS ?= ""
+
+python __anonymous() {
+ import pwd
+ d.setVar('SCHROOT_USER', pwd.getpwuid(os.geteuid()).pw_name)
+ d.setVar('SCHROOT_USER_HOME', pwd.getpwuid(os.geteuid()).pw_dir)
+
+ mode = d.getVar('ISAR_CROSS_COMPILE', True)
+ distro_arch = d.getVar('DISTRO_ARCH')
+ if mode == "0" or d.getVar('HOST_ARCH') == distro_arch or \
+ (d.getVar('HOST_DISTRO') == "debian-stretch" and distro_arch == "i386"):
+ d.setVar('SBUILD_HOST_ARCH', distro_arch)
+ d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_TARGET_DIR'))
+ dep = "sbuild-chroot-target:do_build"
+ else:
+ d.setVar('SBUILD_HOST_ARCH', d.getVar('HOST_ARCH'))
+ d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_HOST_DIR'))
+ dep = "sbuild-chroot-host:do_build"
+ d.setVar('SCHROOT_DEP', dep)
+}
+
+SBUILD_CHROOT ?= "${DEBDISTRONAME}-${SCHROOT_USER}-${@os.getpid()}"
+
+SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}"
+SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}"
+
+schroot_create_configs() {
+ sudo -s <<'EOSUDO'
+ set -e
+
+ cat << EOF > "${SCHROOT_CONF_FILE}"
+[${SBUILD_CHROOT}]
+type=directory
+directory=${SCHROOT_DIR}
+profile=${SBUILD_CHROOT}
+users=${SCHROOT_USER}
+groups=root,sbuild
+root-users=${SCHROOT_USER}
+root-groups=root,sbuild
+source-root-users=${SCHROOT_USER}
+source-root-groups=root,sbuild
+union-type=overlay
+preserve-environment=true
+EOF
+
+ # Prepare mount points
+ cp -rf "${SCHROOT_CONF}/sbuild" "${SBUILD_CONF_DIR}"
+ sbuild_fstab="${SBUILD_CONF_DIR}/fstab"
+
+ fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind 0 0"
+ grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}
+
+ if [ -d ${DL_DIR} ]; then
+ fstab_downloads="${DL_DIR} /downloads none rw,bind 0 0"
+ grep -qxF "${fstab_downloads}" ${sbuild_fstab} || echo "${fstab_downloads}" >> ${sbuild_fstab}
+ fi
+EOSUDO
+}
+
+schroot_delete_configs() {
+ sudo -s <<'EOSUDO'
+ set -e
+ if [ -d "${SBUILD_CONF_DIR}" ]; then
+ rm -rf "${SBUILD_CONF_DIR}"
+ fi
+ rm -f "${SCHROOT_CONF_FILE}"
+EOSUDO
+}
+
+sbuild_export() {
+ SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
+ VAR=${1}; shift
+ VAR_LINE="'${VAR}' => '${@}',"
+ if [ -s "${SBUILD_CONFIG}" ]; then
+ sed -i -e "\$i\\" -e "${VAR_LINE}" ${SBUILD_CONFIG}
+ else
+ echo "\$build_environment = {" > ${SBUILD_CONFIG}
+ echo "${VAR_LINE}" >> ${SBUILD_CONFIG}
+ echo "};" >> ${SBUILD_CONFIG}
+ fi
+ export SBUILD_CONFIG="${SBUILD_CONFIG}"
+}
+
+insert_mounts() {
+ sudo -s <<'EOSUDO'
+ set -e
+ for mp in ${SCHROOT_MOUNTS}; do
+ FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
+ grep -qxF "${FSTAB_LINE}" ${SBUILD_CONF_DIR}/fstab || \
+ echo "${FSTAB_LINE}" >> ${SBUILD_CONF_DIR}/fstab
+ done
+EOSUDO
+}
+
+remove_mounts() {
+ sudo -s <<'EOSUDO'
+ set -e
+ for mp in ${SCHROOT_MOUNTS}; do
+ FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
+ sed -i "\|${FSTAB_LINE}|d" ${SBUILD_CONF_DIR}/fstab
+ done
+EOSUDO
+}
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index a05bd90..cbd9036 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -58,6 +58,8 @@ SSTATE_DIR ?= "${TOPDIR}/sstate-cache"
SSTATE_MANIFESTS = "${TMPDIR}/sstate-control/${DISTRO}-${DISTRO_ARCH}"
BUILDCHROOT_HOST_DIR = "${DEPLOY_DIR_BUILDCHROOT}-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
BUILDCHROOT_TARGET_DIR = "${DEPLOY_DIR_BUILDCHROOT}-target/${DISTRO}-${DISTRO_ARCH}"
+SCHROOT_HOST_DIR = "${DEPLOY_DIR}/schroot-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
+SCHROOT_TARGET_DIR = "${DEPLOY_DIR}/schroot-target/${DISTRO}-${DISTRO_ARCH}"
SDKCHROOT_DIR = "${DEPLOY_DIR_SDKCHROOT}/${DISTRO}-${DISTRO_ARCH}"
CACHE = "${TMPDIR}/cache"
KERNEL_FILE ?= "vmlinuz"
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH v11 05/21] dpkg: Build packages with sbuild
2022-05-23 14:35 [PATCH v11 00/21] Sbuild/Schroot migration Anton Mikanovich
` (3 preceding siblings ...)
2022-05-23 14:36 ` [PATCH v11 04/21] sbuild: Introduce a class for another build method Anton Mikanovich
@ 2022-05-23 14:36 ` Anton Mikanovich
2022-05-23 14:36 ` [PATCH v11 06/21] sbuild: Support of DEB_BUILD_PROFILES Anton Mikanovich
` (16 subsequent siblings)
21 siblings, 0 replies; 23+ messages in thread
From: Anton Mikanovich @ 2022-05-23 14:36 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich, Uladzimir Bely
Use previously exported to schroot rootfs to build the package with
sbuild tool. Provide isar-apt as extra repository to be automatically
add to apt sources.
Also added /home/.git-downloads mount while it's used
as git alternates location.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
meta/classes/dpkg-base.bbclass | 31 ++++++++++++++++---------------
meta/classes/dpkg-gbp.bbclass | 2 ++
meta/classes/dpkg.bbclass | 10 ++++++++--
meta/classes/sbuild.bbclass | 4 ++--
4 files changed, 28 insertions(+), 19 deletions(-)
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 7e032ba..dbe35f4 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -4,6 +4,7 @@
#
# SPDX-License-Identifier: MIT
+inherit sbuild
inherit buildchroot
inherit debianize
inherit terminal
@@ -16,6 +17,8 @@ DEPENDS_append_riscv64 = "${@' crossbuild-essential-riscv64' if d.getVar('ISAR_C
DEB_BUILD_PROFILES ?= ""
DEB_BUILD_OPTIONS ?= ""
+ISAR_APT_REPO ?= "deb [trusted=yes] file:///home/builder/${PN}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO} ${DEBDISTRONAME} main"
+
python do_adjust_git() {
import subprocess
@@ -175,6 +178,7 @@ addtask prepare_build after do_patch do_transform_template before do_dpkg_build
# If Isar recipes depend on each other, they typically need the package
# deployed to isar-apt
do_prepare_build[deptask] = "do_deploy_deb"
+do_prepare_build[depends] = "${SCHROOT_DEP}"
BUILDROOT = "${BUILDCHROOT_DIR}/${PP}"
@@ -200,6 +204,15 @@ dpkg_undo_mounts() {
sudo rmdir ${BUILDROOT}
}
+do_prepare_build_append() {
+ # Make a local copy of isar-apt repo that is not affected by other parallel builds
+ mkdir -p ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
+ rm -rf ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/*
+ cp -Rl ${REPO_ISAR_DIR} ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
+}
+
+do_prepare_build[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
+
# Placeholder for actual dpkg_runbuild() implementation
dpkg_runbuild() {
die "This should never be called, overwrite it in your derived class"
@@ -222,14 +235,11 @@ def isar_export_build_settings(d):
os.environ['DEB_BUILD_PROFILES'] = isar_deb_build_profiles(d)
python do_dpkg_build() {
- lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
- shared=True)
- bb.build.exec_func("dpkg_do_mounts", d)
+ bb.build.exec_func('schroot_create_configs', d)
try:
bb.build.exec_func("dpkg_runbuild", d)
finally:
- bb.build.exec_func("dpkg_undo_mounts", d)
- bb.utils.unlockfile(lock)
+ bb.build.exec_func('schroot_delete_configs', d)
}
addtask dpkg_build
@@ -259,7 +269,7 @@ python do_dpkg_build_setscene() {
addtask dpkg_build_setscene
do_dpkg_build_setscene[dirs] += "${S}/.."
-KEEP_INSTALLED_ON_CLEAN ?= "0"
+do_dpkg_build[depends] = "${SCHROOT_DEP}"
CLEANFUNCS += "deb_clean"
@@ -269,15 +279,6 @@ deb_clean() {
for d in ${DEBS}; do
repo_del_package "${REPO_ISAR_DIR}"/"${DISTRO}" \
"${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${d}"
- if [ "${KEEP_INSTALLED_ON_CLEAN}" = "1" ]; then
- continue;
- fi
- package=$(basename "${d}")
- package_remove="/usr/bin/apt-get remove -y ${package%%_*}"
- sudo -E chroot ${BUILDCHROOT_DIR} ${package_remove} || true
- if [ "${BUILDCHROOT_DIR}" != "${BUILDCHROOT_TARGET_DIR}" ]; then
- sudo -E chroot ${BUILDCHROOT_TARGET_DIR} ${package_remove} || true
- fi
done
fi
}
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index 2b0b818..68172b3 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -11,6 +11,8 @@ PATCHTOOL ?= "git"
GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
+SCHROOT_MOUNTS = "${WORKDIR}:${PP} ${GITDIR}:/home/.git-downloads"
+
dpkg_runbuild_prepend() {
sh -c "
cd ${WORKDIR}/${PPS}
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index af83353..5f68fd6 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -39,6 +39,12 @@ dpkg_runbuild() {
export DEB_BUILD_OPTIONS="${@ isar_deb_build_options(d)}"
export DEB_BUILD_PROFILES="${@ isar_deb_build_profiles(d)}"
export PARALLEL_MAKE="${PARALLEL_MAKE}"
- sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
- /isar/build.sh ${PP}/${PPS} ${PACKAGE_ARCH}
+
+ export SBUILD_CONFIG="${SBUILD_CONFIG}"
+
+ sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
+ --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \
+ --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
+ --debbuildopts="--source-option=-I" \
+ --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
}
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index b9fc6a0..a3783bc 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -29,6 +29,8 @@ SBUILD_CHROOT ?= "${DEBDISTRONAME}-${SCHROOT_USER}-${@os.getpid()}"
SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}"
SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}"
+SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
+
schroot_create_configs() {
sudo -s <<'EOSUDO'
set -e
@@ -73,7 +75,6 @@ EOSUDO
}
sbuild_export() {
- SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
VAR=${1}; shift
VAR_LINE="'${VAR}' => '${@}',"
if [ -s "${SBUILD_CONFIG}" ]; then
@@ -83,7 +84,6 @@ sbuild_export() {
echo "${VAR_LINE}" >> ${SBUILD_CONFIG}
echo "};" >> ${SBUILD_CONFIG}
fi
- export SBUILD_CONFIG="${SBUILD_CONFIG}"
}
insert_mounts() {
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH v11 06/21] sbuild: Support of DEB_BUILD_PROFILES
2022-05-23 14:35 [PATCH v11 00/21] Sbuild/Schroot migration Anton Mikanovich
` (4 preceding siblings ...)
2022-05-23 14:36 ` [PATCH v11 05/21] dpkg: Build packages with sbuild Anton Mikanovich
@ 2022-05-23 14:36 ` Anton Mikanovich
2022-05-23 14:36 ` [PATCH v11 07/21] sbuild: Support of shell exports from dpkg_runbuild_prepend Anton Mikanovich
` (15 subsequent siblings)
21 siblings, 0 replies; 23+ messages in thread
From: Anton Mikanovich @ 2022-05-23 14:36 UTC (permalink / raw)
To: isar-users; +Cc: Uladzimir Bely
From: Uladzimir Bely <ubely@ilbers.de>
In Isar packages may use their own profiles. For example, linux-custom.inc
introduces 'nolibcdev' profile that should be passed to dpkg-buildpackage
in both cross and native builds.
By default, sbuild for cross-building uses passes "cross,nocheck" options
to dpkg-buildpackage and ignores DEB_BUILD_PROFILES environment variable.
This change makes sbuild use custom profiles even in cross-build mode.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
meta/classes/dpkg.bbclass | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 5f68fd6..218c7e8 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -37,13 +37,17 @@ dpkg_runbuild() {
E="${@ isar_export_proxies(d)}"
E="${@ isar_export_ccache(d)}"
export DEB_BUILD_OPTIONS="${@ isar_deb_build_options(d)}"
- export DEB_BUILD_PROFILES="${@ isar_deb_build_profiles(d)}"
export PARALLEL_MAKE="${PARALLEL_MAKE}"
+ profiles="${@ isar_deb_build_profiles(d)}"
+ if [ ! -z "$profiles" ]; then
+ profiles=$(echo --profiles="$profiles" | sed -e 's/ \+/,/g')
+ fi
+
export SBUILD_CONFIG="${SBUILD_CONFIG}"
sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
- --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \
+ --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
--no-run-lintian --no-run-piuparts --no-run-autopkgtest \
--debbuildopts="--source-option=-I" \
--build-dir=${WORKDIR} ${WORKDIR}/${PPS}
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH v11 07/21] sbuild: Support of shell exports from dpkg_runbuild_prepend
2022-05-23 14:35 [PATCH v11 00/21] Sbuild/Schroot migration Anton Mikanovich
` (5 preceding siblings ...)
2022-05-23 14:36 ` [PATCH v11 06/21] sbuild: Support of DEB_BUILD_PROFILES Anton Mikanovich
@ 2022-05-23 14:36 ` Anton Mikanovich
2022-05-23 14:36 ` [PATCH v11 08/21] dpkg: Remove builddeps install task Anton Mikanovich
` (14 subsequent siblings)
21 siblings, 0 replies; 23+ messages in thread
From: Anton Mikanovich @ 2022-05-23 14:36 UTC (permalink / raw)
To: isar-users; +Cc: Uladzimir Bely
From: Uladzimir Bely <ubely@ilbers.de>
Many of recipes often use shell exports done in dpkg_run_prepend.
While sbuild is performed in isolated environment, we need a way to pass
these exported variables to it. This is done by storing environment just
before dpkg_runbuild (after prepare_build) and finding just before
the actual build what has been changed or added.
This patch allows using shell exports, but warning will be produced.
It's recommended to modify the code (in downstreams) and replace shell
exports by other methods, like using templates.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
meta/classes/dpkg.bbclass | 26 ++++++++++++++++++++++++++
1 file changed, 26 insertions(+)
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 218c7e8..05b46c9 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -32,6 +32,12 @@ do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
addtask devshell after do_install_builddeps
+DPKG_PREBUILD_ENV_FILE="${WORKDIR}/dpkg_prebuild.env"
+
+do_prepare_build_append() {
+ env > ${DPKG_PREBUILD_ENV_FILE}
+}
+
# Build package from sources using build script
dpkg_runbuild() {
E="${@ isar_export_proxies(d)}"
@@ -39,6 +45,26 @@ dpkg_runbuild() {
export DEB_BUILD_OPTIONS="${@ isar_deb_build_options(d)}"
export PARALLEL_MAKE="${PARALLEL_MAKE}"
+ env | while read -r line; do
+ # Filter the same lines
+ grep -q "^${line}" ${DPKG_PREBUILD_ENV_FILE} && continue
+ # Filter some standard variables
+ echo ${line} | grep -q "^HOME=" && continue
+ echo ${line} | grep -q "^PWD=" && continue
+
+ var=$(echo "${line}" | cut -d '=' -f1)
+ value=$(echo "${line}" | cut -d '=' -f2-)
+ sbuild_export $var "$value"
+
+ # Don't warn some variables
+ [ "${var}" = "PARALLEL_MAKE" ] && continue
+ [ "${var}" = "CCACHE_DIR" ] && continue
+ [ "${var}" = "PATH_PREPEND" ] && continue
+ [ "${var}" = "DEB_BUILD_OPTIONS" ] && continue
+
+ bbwarn "Export of '${line}' detected, please migrate to templates"
+ done
+
profiles="${@ isar_deb_build_profiles(d)}"
if [ ! -z "$profiles" ]; then
profiles=$(echo --profiles="$profiles" | sed -e 's/ \+/,/g')
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH v11 08/21] dpkg: Remove builddeps install task
2022-05-23 14:35 [PATCH v11 00/21] Sbuild/Schroot migration Anton Mikanovich
` (6 preceding siblings ...)
2022-05-23 14:36 ` [PATCH v11 07/21] sbuild: Support of shell exports from dpkg_runbuild_prepend Anton Mikanovich
@ 2022-05-23 14:36 ` Anton Mikanovich
2022-05-23 14:36 ` [PATCH v11 09/21] sbuild: Add ccache support Anton Mikanovich
` (13 subsequent siblings)
21 siblings, 0 replies; 23+ messages in thread
From: Anton Mikanovich @ 2022-05-23 14:36 UTC (permalink / raw)
To: isar-users; +Cc: Uladzimir Bely, Anton Mikanovich
From: Uladzimir Bely <ubely@ilbers.de>
While builddeps are now handled interlnally by sbuild, we don't need
a task to preinstall them in buildchroot.
But for the second local (base-apt) build we need a way to keep
all dependencies in form of .deb files in DL_DIR. This is done
by executing additional commands in sbuild that copy them to/from
an externally mounted folder.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
meta/classes/dpkg.bbclass | 47 ++++++++++++++++---------------------
meta/classes/sbuild.bbclass | 1 -
2 files changed, 20 insertions(+), 28 deletions(-)
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 05b46c9..3c7b670 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -5,33 +5,6 @@ inherit dpkg-base
PACKAGE_ARCH ?= "${DISTRO_ARCH}"
-# Install build dependencies for package
-do_install_builddeps() {
- dpkg_do_mounts
- E="${@ isar_export_proxies(d)}"
- export DEB_BUILD_OPTIONS="${@ isar_deb_build_options(d)}"
- export DEB_BUILD_PROFILES="${@ isar_deb_build_profiles(d)}"
- distro="${DISTRO}"
- if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
- distro="${HOST_DISTRO}"
- fi
-
- deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
- sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
- ${PP}/${PPS} ${PACKAGE_ARCH} --download-only
- deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
- sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
- ${PP}/${PPS} ${PACKAGE_ARCH}
- dpkg_undo_mounts
-}
-
-addtask install_builddeps after do_prepare_build before do_dpkg_build
-do_install_builddeps[depends] += "${BUILDCHROOT_DEP} isar-apt:do_cache_config"
-# apt and reprepro may not run in parallel, acquire the Isar lock
-do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
-
-addtask devshell after do_install_builddeps
-
DPKG_PREBUILD_ENV_FILE="${WORKDIR}/dpkg_prebuild.env"
do_prepare_build_append() {
@@ -65,6 +38,16 @@ dpkg_runbuild() {
bbwarn "Export of '${line}' detected, please migrate to templates"
done
+ distro="${DISTRO}"
+ if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+ distro="${HOST_DISTRO}"
+ fi
+
+ deb_dl_dir_import "${WORKDIR}/rootfs" "${distro}"
+
+ deb_dir="/var/cache/apt/archives"
+ ext_deb_dir="${PP}/rootfs/${deb_dir}"
+
profiles="${@ isar_deb_build_profiles(d)}"
if [ ! -z "$profiles" ]; then
profiles=$(echo --profiles="$profiles" | sed -e 's/ \+/,/g')
@@ -72,9 +55,19 @@ dpkg_runbuild() {
export SBUILD_CONFIG="${SBUILD_CONFIG}"
+ echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
+
sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
--host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
--no-run-lintian --no-run-piuparts --no-run-autopkgtest \
+ --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
+ --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
+ --finished-build-commands="cp -n --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
--debbuildopts="--source-option=-I" \
--build-dir=${WORKDIR} ${WORKDIR}/${PPS}
+
+ deb_dl_dir_export "${WORKDIR}/rootfs" "${distro}"
+
+ # Cleanup apt artifacts
+ sudo rm -rf ${WORKDIR}/rootfs
}
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index a3783bc..178ba1b 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -8,7 +8,6 @@ SCHROOT_MOUNTS ?= ""
python __anonymous() {
import pwd
d.setVar('SCHROOT_USER', pwd.getpwuid(os.geteuid()).pw_name)
- d.setVar('SCHROOT_USER_HOME', pwd.getpwuid(os.geteuid()).pw_dir)
mode = d.getVar('ISAR_CROSS_COMPILE', True)
distro_arch = d.getVar('DISTRO_ARCH')
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH v11 09/21] sbuild: Add ccache support
2022-05-23 14:35 [PATCH v11 00/21] Sbuild/Schroot migration Anton Mikanovich
` (7 preceding siblings ...)
2022-05-23 14:36 ` [PATCH v11 08/21] dpkg: Remove builddeps install task Anton Mikanovich
@ 2022-05-23 14:36 ` Anton Mikanovich
2022-05-23 14:36 ` [PATCH v11 10/21] dpkg-base: Switch devshell to use schroot Anton Mikanovich
` (12 subsequent siblings)
21 siblings, 0 replies; 23+ messages in thread
From: Anton Mikanovich @ 2022-05-23 14:36 UTC (permalink / raw)
To: isar-users; +Cc: Uladzimir Bely
From: Uladzimir Bely <ubely@ilbers.de>
This adds ccache support for custom packages in the same manner
as it was done previously for buildchroot-based build, by using
USE_CCACHE variable in local.conf or per-recipe.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
meta/classes/dpkg.bbclass | 4 ++++
meta/classes/sbuild.bbclass | 23 +++++++++++++++++++
.../sbuild-chroot/sbuild-chroot.inc | 1 +
3 files changed, 28 insertions(+)
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 3c7b670..d92135c 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -48,6 +48,10 @@ dpkg_runbuild() {
deb_dir="/var/cache/apt/archives"
ext_deb_dir="${PP}/rootfs/${deb_dir}"
+ if [ ${USE_CCACHE} -eq 1 ]; then
+ schroot_configure_ccache
+ fi
+
profiles="${@ isar_deb_build_profiles(d)}"
if [ ! -z "$profiles" ]; then
profiles=$(echo --profiles="$profiles" | sed -e 's/ \+/,/g')
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 178ba1b..6ff41dd 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -105,3 +105,26 @@ remove_mounts() {
done
EOSUDO
}
+
+schroot_configure_ccache() {
+ sudo -s <<'EOSUDO'
+ set -e
+
+ sbuild_fstab="${SBUILD_CONF_DIR}/fstab"
+
+ install --group=sbuild --mode=2775 -d ${CCACHE_DIR}
+ fstab_ccachedir="${CCACHE_DIR} /ccache none rw,bind 0 0"
+ grep -qxF "${fstab_ccachedir}" ${sbuild_fstab} || echo "${fstab_ccachedir}" >> ${sbuild_fstab}
+
+ (flock 9
+ [ -w ${CCACHE_DIR}/sbuild-setup ] || cat << END > ${CCACHE_DIR}/sbuild-setup
+#!/bin/sh
+export PATH="\$PATH_PREPEND:\$PATH"
+exec "\$@"
+END
+ chmod a+rx ${CCACHE_DIR}/sbuild-setup
+ ) 9>"${CCACHE_DIR}/sbuild-setup.lock"
+
+ echo "command-prefix=/ccache/sbuild-setup" >> "${SCHROOT_CONF_FILE}"
+EOSUDO
+}
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
index 177a8a6..74811e2 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -14,6 +14,7 @@ SBUILD_CHROOT_PREINSTALL_COMMON = " \
fakeroot \
build-essential \
debhelper \
+ ccache \
"
SBUILD_CHROOT_PREINSTALL ?= " \
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH v11 10/21] dpkg-base: Switch devshell to use schroot
2022-05-23 14:35 [PATCH v11 00/21] Sbuild/Schroot migration Anton Mikanovich
` (8 preceding siblings ...)
2022-05-23 14:36 ` [PATCH v11 09/21] sbuild: Add ccache support Anton Mikanovich
@ 2022-05-23 14:36 ` Anton Mikanovich
2022-05-23 14:36 ` [PATCH v11 11/21] dpkg-base: Switch apt_fetch and apt_unpack " Anton Mikanovich
` (11 subsequent siblings)
21 siblings, 0 replies; 23+ messages in thread
From: Anton Mikanovich @ 2022-05-23 14:36 UTC (permalink / raw)
To: isar-users; +Cc: Uladzimir Bely
From: Uladzimir Bely <ubely@ilbers.de>
While packages are now build in schroot with sbuild, devshell
task should make terminal open inside sbuild environment.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
meta/classes/dpkg-base.bbclass | 23 ++++++++++++++-----
meta/classes/sbuild.bbclass | 3 +++
.../sbuild-chroot/sbuild-chroot.inc | 2 ++
3 files changed, 22 insertions(+), 6 deletions(-)
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index dbe35f4..7005aa5 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -298,19 +298,30 @@ do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
do_deploy_deb[dirs] = "${S}"
python do_devshell() {
- bb.build.exec_func('dpkg_do_mounts', d)
+ bb.build.exec_func('schroot_create_configs', d)
isar_export_proxies(d)
isar_export_ccache(d)
isar_export_build_settings(d)
- buildchroot = d.getVar('BUILDCHROOT_DIR')
+ schroot = d.getVar('SBUILD_CHROOT')
+ isar_apt = d.getVar('ISAR_APT_REPO')
pp_pps = os.path.join(d.getVar('PP'), d.getVar('PPS'))
- # the PATH variable is not forwarded by sudo -E.
- termcmd = "sudo -E chroot {0} sh -c 'cd {1}; export PATH=$PATH_PREPEND:$PATH; $SHELL -i'"
- oe_terminal(termcmd.format(buildchroot, pp_pps), "Isar devshell", d)
- bb.build.exec_func('dpkg_undo_mounts', d)
+ install_deps = ":" if d.getVar('BB_CURRENTTASK') == "devshell_nodeps" else "mk-build-deps -i -t \
+ \"apt-get -y -q -o Debug::pkgProblemResolver=yes --no-install-recommends --allow-downgrades\" \
+ debian/control"
+
+ termcmd = "schroot -d / -c {0} -u root -- sh -c ' \
+ cd {1}; \
+ echo {2} > /etc/apt/sources.list.d/isar_apt.list; \
+ apt-get -y -q update; \
+ {3}; \
+ $SHELL -i \
+ '"
+ oe_terminal(termcmd.format(schroot, pp_pps, isar_apt, install_deps), "Isar devshell", d)
+
+ bb.build.exec_func('schroot_delete_configs', d)
}
addtask devshell after do_prepare_build
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 6ff41dd..2d52307 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -56,6 +56,9 @@ EOF
fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind 0 0"
grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}
+ fstab_pkgdir="${WORKDIR} /home/builder/${PN} none rw,bind 0 0"
+ grep -qxF "${fstab_pkgdir}" ${sbuild_fstab} || echo "${fstab_pkgdir}" >> ${sbuild_fstab}
+
if [ -d ${DL_DIR} ]; then
fstab_downloads="${DL_DIR} /downloads none rw,bind 0 0"
grep -qxF "${fstab_downloads}" ${sbuild_fstab} || echo "${fstab_downloads}" >> ${sbuild_fstab}
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
index 74811e2..4688da1 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -15,6 +15,8 @@ SBUILD_CHROOT_PREINSTALL_COMMON = " \
build-essential \
debhelper \
ccache \
+ devscripts \
+ equivs \
"
SBUILD_CHROOT_PREINSTALL ?= " \
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH v11 11/21] dpkg-base: Switch apt_fetch and apt_unpack to use schroot
2022-05-23 14:35 [PATCH v11 00/21] Sbuild/Schroot migration Anton Mikanovich
` (9 preceding siblings ...)
2022-05-23 14:36 ` [PATCH v11 10/21] dpkg-base: Switch devshell to use schroot Anton Mikanovich
@ 2022-05-23 14:36 ` Anton Mikanovich
2022-05-23 14:36 ` [PATCH v11 12/21] doc: Add sbuild-related documentation Anton Mikanovich
` (10 subsequent siblings)
21 siblings, 0 replies; 23+ messages in thread
From: Anton Mikanovich @ 2022-05-23 14:36 UTC (permalink / raw)
To: isar-users; +Cc: Uladzimir Bely
From: Uladzimir Bely <ubely@ilbers.de>
This moves downloading and unpacking deb-src to schroot environment.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
meta/classes/dpkg-base.bbclass | 25 +++++++++----------------
1 file changed, 9 insertions(+), 16 deletions(-)
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 7005aa5..87dc3f7 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -104,19 +104,12 @@ python() {
}
do_apt_fetch() {
- dpkg_do_mounts
- E="${@ isar_export_proxies(d)}"
- sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update \
- -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
- -o Dir::Etc::SourceParts="-" \
- -o APT::Get::List-Cleanup="0"
-
+ schroot_create_configs
for uri in "${SRC_APT}"; do
- sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+ schroot -d / -c ${SBUILD_CHROOT} -- \
sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only-source source "$2"' my_script "${DISTRO}" "${uri}"
done
-
- dpkg_undo_mounts
+ schroot_delete_configs
}
addtask apt_fetch
@@ -125,13 +118,14 @@ do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
# Add dependency from the correct buildchroot: host or target
do_apt_fetch[depends] += "${BUILDCHROOT_DEP}"
+# Add dependency from the correct schroot: host or target
+do_apt_fetch[depends] += "${SCHROOT_DEP}"
+
do_apt_unpack() {
rm -rf ${S}
- dpkg_do_mounts
- E="${@ isar_export_proxies(d)}"
-
+ schroot_create_configs
for uri in "${SRC_APT}"; do
- sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+ schroot -d / -c ${SBUILD_CHROOT} -- \
sh -c ' \
set -e
dscfile="$(apt-get -y -qq --print-uris --only-source source "${2}" | cut -d " " -f2 | grep -E "*.dsc")"
@@ -140,8 +134,7 @@ do_apt_unpack() {
dpkg-source -x "${dscfile}" "${PPS}"' \
my_script "${DISTRO}" "${uri}"
done
-
- dpkg_undo_mounts
+ schroot_delete_configs
}
addtask apt_unpack after do_apt_fetch
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH v11 12/21] doc: Add sbuild-related documentation
2022-05-23 14:35 [PATCH v11 00/21] Sbuild/Schroot migration Anton Mikanovich
` (10 preceding siblings ...)
2022-05-23 14:36 ` [PATCH v11 11/21] dpkg-base: Switch apt_fetch and apt_unpack " Anton Mikanovich
@ 2022-05-23 14:36 ` Anton Mikanovich
2022-05-23 14:36 ` [PATCH v11 13/21] sbuild: Use .dsc file instead of source directory Anton Mikanovich
` (9 subsequent siblings)
21 siblings, 0 replies; 23+ messages in thread
From: Anton Mikanovich @ 2022-05-23 14:36 UTC (permalink / raw)
To: isar-users; +Cc: Uladzimir Bely
From: Uladzimir Bely <ubely@ilbers.de>
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
doc/user_manual.md | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
diff --git a/doc/user_manual.md b/doc/user_manual.md
index bcff8d9..ee68f8b 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -86,6 +86,24 @@ apt install \
pristine-tar
```
+Additional setup is required since `sbuild` is now used for package build.
+Install the following packages:
+```
+apt install \
+ sbuild \
+ schroot
+```
+Also, user who runs isar should be added to `sbuild` group.
+
+**NOTE:** sbuild version (<=0.78.1) packaged in Debian Buster doesn't support
+`$apt_keep_downloaded_packages` option which is required in Isar for
+populating `${DL_DIR}/deb`. So, host `sbuild` in this case should be manually
+upgraded to >=0.81.2 version from Debian Bullseye.
+
+```
+sudo gpasswd -a <username> sbuild
+```
+
If your host is >= buster, also install the following package.
```
apt install python3-distutils
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH v11 13/21] sbuild: Use .dsc file instead of source directory
2022-05-23 14:35 [PATCH v11 00/21] Sbuild/Schroot migration Anton Mikanovich
` (11 preceding siblings ...)
2022-05-23 14:36 ` [PATCH v11 12/21] doc: Add sbuild-related documentation Anton Mikanovich
@ 2022-05-23 14:36 ` Anton Mikanovich
2022-05-23 14:36 ` [PATCH v11 14/21] sbuild: Fixed proxy support Anton Mikanovich
` (8 subsequent siblings)
21 siblings, 0 replies; 23+ messages in thread
From: Anton Mikanovich @ 2022-05-23 14:36 UTC (permalink / raw)
To: isar-users; +Cc: Uladzimir Bely
From: Uladzimir Bely <ubely@ilbers.de>
This makes build more Debian-like and avoids additional execution of
"dh clean" by sbuild before dependencies resolved and build started.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
meta/classes/dpkg.bbclass | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index d92135c..564e654 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -61,6 +61,10 @@ dpkg_runbuild() {
echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
+ # Create a .dsc file from source directory to use it with sbuild
+ sh -c "cd ${WORKDIR}; dpkg-source -q -b ${PPS}"
+ DSC=$(head -n1 ${WORKDIR}/${PPS}/debian/changelog | awk '{gsub(/[()]/,""); printf "%s_%s.dsc", $1, $2}')
+
sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
--host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
--no-run-lintian --no-run-piuparts --no-run-autopkgtest \
@@ -68,7 +72,7 @@ dpkg_runbuild() {
--finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
--finished-build-commands="cp -n --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
--debbuildopts="--source-option=-I" \
- --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
+ --build-dir=${WORKDIR} --dist="isar" ${WORKDIR}/${DSC}
deb_dl_dir_export "${WORKDIR}/rootfs" "${distro}"
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH v11 14/21] sbuild: Fixed proxy support
2022-05-23 14:35 [PATCH v11 00/21] Sbuild/Schroot migration Anton Mikanovich
` (12 preceding siblings ...)
2022-05-23 14:36 ` [PATCH v11 13/21] sbuild: Use .dsc file instead of source directory Anton Mikanovich
@ 2022-05-23 14:36 ` Anton Mikanovich
2022-05-23 14:36 ` [PATCH v11 15/21] sbuild: Fix debsrc_download for packages dependencies Anton Mikanovich
` (7 subsequent siblings)
21 siblings, 0 replies; 23+ messages in thread
From: Anton Mikanovich @ 2022-05-23 14:36 UTC (permalink / raw)
To: isar-users; +Cc: Uladzimir Bely
From: Uladzimir Bely <ubely@ilbers.de>
This makes sbuild able to work via proxy.
Proxy settings passed via 'sbuild_export' seem to work only for build
stage in sbuild. Previous stages (like schroot setup, downloading deps)
don't use this values.
To work via proxy in this stages there should be environment filter
for sbuild set via sbuild.conf.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
meta/classes/dpkg-base.bbclass | 1 +
meta/classes/dpkg.bbclass | 14 ++++++++++++++
meta/classes/sbuild.bbclass | 31 ++++++++++++++++++++++++-------
3 files changed, 39 insertions(+), 7 deletions(-)
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 87dc3f7..7948b23 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -104,6 +104,7 @@ python() {
}
do_apt_fetch() {
+ E="${@ isar_export_proxies(d)}"
schroot_create_configs
for uri in "${SRC_APT}"; do
schroot -d / -c ${SBUILD_CHROOT} -- \
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 564e654..d8c4dd8 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -35,6 +35,15 @@ dpkg_runbuild() {
[ "${var}" = "PATH_PREPEND" ] && continue
[ "${var}" = "DEB_BUILD_OPTIONS" ] && continue
+ [ "${var}" = "http_proxy" ] && continue
+ [ "${var}" = "HTTP_PROXY" ] && continue
+ [ "${var}" = "https_proxy" ] && continue
+ [ "${var}" = "HTTPS_PROXY" ] && continue
+ [ "${var}" = "ftp_proxy" ] && continue
+ [ "${var}" = "FTP_PROXY" ] && continue
+ [ "${var}" = "no_proxy" ] && continue
+ [ "${var}" = "NO_PROXY" ] && continue
+
bbwarn "Export of '${line}' detected, please migrate to templates"
done
@@ -59,6 +68,11 @@ dpkg_runbuild() {
export SBUILD_CONFIG="${SBUILD_CONFIG}"
+ for envvar in http_proxy HTTP_PROXY https_proxy HTTPS_PROXY \
+ ftp_proxy FTP_PROXY no_proxy NO_PROXY; do
+ sbuild_add_env_filter "$envvar"
+ done
+
echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
# Create a .dsc file from source directory to use it with sbuild
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 2d52307..7d24c47 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -76,16 +76,33 @@ schroot_delete_configs() {
EOSUDO
}
+sbuild_add_env_filter() {
+ [ -w ${SBUILD_CONFIG} ] || touch ${SBUILD_CONFIG}
+
+ if ! grep -q "^\$environment_filter =" ${SBUILD_CONFIG}; then
+ echo "\$environment_filter = [" >> ${SBUILD_CONFIG}
+ echo "];" >> ${SBUILD_CONFIG}
+ fi
+
+ FILTER=${1}
+
+ sed -i -e "/'\^${FILTER}\\$/d" \
+ -e "/^\$environment_filter =.*/a '^${FILTER}\$'," ${SBUILD_CONFIG}
+}
+
sbuild_export() {
- VAR=${1}; shift
- VAR_LINE="'${VAR}' => '${@}',"
- if [ -s "${SBUILD_CONFIG}" ]; then
- sed -i -e "\$i\\" -e "${VAR_LINE}" ${SBUILD_CONFIG}
- else
- echo "\$build_environment = {" > ${SBUILD_CONFIG}
- echo "${VAR_LINE}" >> ${SBUILD_CONFIG}
+ [ -w ${SBUILD_CONFIG} ] || touch ${SBUILD_CONFIG}
+
+ if ! grep -q "^\$build_environment =" ${SBUILD_CONFIG}; then
+ echo "\$build_environment = {" >> ${SBUILD_CONFIG}
echo "};" >> ${SBUILD_CONFIG}
fi
+
+ VAR=${1}; shift
+ VAR_LINE="'${VAR}' => '${@}',"
+
+ sed -i -e "/^'${VAR}' =>/d" ${SBUILD_CONFIG} \
+ -e "/^\$build_environment =.*/a ${VAR_LINE}" ${SBUILD_CONFIG}
}
insert_mounts() {
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH v11 15/21] sbuild: Fix debsrc_download for packages dependencies
2022-05-23 14:35 [PATCH v11 00/21] Sbuild/Schroot migration Anton Mikanovich
` (13 preceding siblings ...)
2022-05-23 14:36 ` [PATCH v11 14/21] sbuild: Fixed proxy support Anton Mikanovich
@ 2022-05-23 14:36 ` Anton Mikanovich
2022-05-23 14:36 ` [PATCH v11 16/21] fix: support build of packages with epoch version Anton Mikanovich
` (6 subsequent siblings)
21 siblings, 0 replies; 23+ messages in thread
From: Anton Mikanovich @ 2022-05-23 14:36 UTC (permalink / raw)
To: isar-users; +Cc: Uladzimir Bely
From: Uladzimir Bely <ubely@ilbers.de>
With sbuild, build deps are internally installed in temporary sbuild
environment,so they don't appear in any external dpkg.log files.
With "cache-deb-src" base repo feature enabled, these such packages
are considered not to belong to current build, so their src packages
are not downloaded at image postprocess stage.
The patch fixes this by extracting dpkg.log files from the sbuild
environment to one common log file. This consolidated log is then
used at rootfs_postprocess to get source packages.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
meta/classes/deb-dl-dir.bbclass | 4 ++++
meta/classes/dpkg.bbclass | 6 +++++-
meta/classes/sbuild.bbclass | 9 +++++++++
3 files changed, 18 insertions(+), 1 deletion(-)
diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 4780228..f479b29 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -14,6 +14,10 @@ is_not_part_of_current_build() {
# build using the below method.
local output="$( grep -xhs ".* status installed ${package}:${arch} ${version}" \
"${IMAGE_ROOTFS}"/var/log/dpkg.log \
+ "${SCHROOT_HOST_DIR}"/var/log/dpkg.log \
+ "${SCHROOT_TARGET_DIR}"/var/log/dpkg.log \
+ "${SCHROOT_HOST_DIR}"/tmp/dpkg_common.log \
+ "${SCHROOT_TARGET_DIR}"/tmp/dpkg_common.log \
"${BUILDCHROOT_HOST_DIR}"/var/log/dpkg.log \
"${BUILDCHROOT_TARGET_DIR}"/var/log/dpkg.log | head -1 )"
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index d8c4dd8..a0544d5 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -55,7 +55,8 @@ dpkg_runbuild() {
deb_dl_dir_import "${WORKDIR}/rootfs" "${distro}"
deb_dir="/var/cache/apt/archives"
- ext_deb_dir="${PP}/rootfs/${deb_dir}"
+ ext_root="${PP}/rootfs"
+ ext_deb_dir="${ext_root}${deb_dir}"
if [ ${USE_CCACHE} -eq 1 ]; then
schroot_configure_ccache
@@ -82,12 +83,15 @@ dpkg_runbuild() {
sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
--host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
--no-run-lintian --no-run-piuparts --no-run-autopkgtest \
+ --chroot-setup-commands="rm -f /var/log/dpkg.log" \
--chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
--finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
--finished-build-commands="cp -n --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
+ --finished-build-commands="cp /var/log/dpkg.log ${ext_root}/dpkg_partial.log" \
--debbuildopts="--source-option=-I" \
--build-dir=${WORKDIR} --dist="isar" ${WORKDIR}/${DSC}
+ sbuild_dpkg_log_export "${WORKDIR}/rootfs/dpkg_partial.log"
deb_dl_dir_export "${WORKDIR}/rootfs" "${distro}"
# Cleanup apt artifacts
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 7d24c47..a78b15b 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -148,3 +148,12 @@ END
echo "command-prefix=/ccache/sbuild-setup" >> "${SCHROOT_CONF_FILE}"
EOSUDO
}
+
+sbuild_dpkg_log_export() {
+ export dpkg_partial_log="${1}"
+
+ ( flock 9
+ set -e
+ cat ${dpkg_partial_log} >> ${SCHROOT_DIR}/tmp/dpkg_common.log
+ ) 9>"${SCHROOT_DIR}/tmp/dpkg_common.log.lock"
+}
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH v11 16/21] fix: support build of packages with epoch version
2022-05-23 14:35 [PATCH v11 00/21] Sbuild/Schroot migration Anton Mikanovich
` (14 preceding siblings ...)
2022-05-23 14:36 ` [PATCH v11 15/21] sbuild: Fix debsrc_download for packages dependencies Anton Mikanovich
@ 2022-05-23 14:36 ` Anton Mikanovich
2022-05-23 14:36 ` [PATCH v11 17/21] always create apt-cache dirs in deb_dl_dir_import Anton Mikanovich
` (5 subsequent siblings)
21 siblings, 0 replies; 23+ messages in thread
From: Anton Mikanovich @ 2022-05-23 14:36 UTC (permalink / raw)
To: isar-users; +Cc: Felix Moessbauer
From: Felix Moessbauer <felix.moessbauer@siemens.com>
This patch reworks the logic how the DSC file for building
with sbuild is located.
By that, we do not rely on implementation details of dpkg-source
regarding the name of the generated .dsc file.
Instead, we remove all <package>*.dsc files in advance, let dpkg-source
generate the file and use a find to locate the generated one.
By that, package versions with epochs (e.g. "1:1.0.0") are
supported.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
meta/classes/dpkg.bbclass | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index a0544d5..3dda7d2 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -77,8 +77,10 @@ dpkg_runbuild() {
echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
# Create a .dsc file from source directory to use it with sbuild
+ DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
+ find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -delete
sh -c "cd ${WORKDIR}; dpkg-source -q -b ${PPS}"
- DSC=$(head -n1 ${WORKDIR}/${PPS}/debian/changelog | awk '{gsub(/[()]/,""); printf "%s_%s.dsc", $1, $2}')
+ DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -print)
sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
--host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
@@ -89,7 +91,7 @@ dpkg_runbuild() {
--finished-build-commands="cp -n --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
--finished-build-commands="cp /var/log/dpkg.log ${ext_root}/dpkg_partial.log" \
--debbuildopts="--source-option=-I" \
- --build-dir=${WORKDIR} --dist="isar" ${WORKDIR}/${DSC}
+ --build-dir=${WORKDIR} --dist="isar" ${DSC_FILE}
sbuild_dpkg_log_export "${WORKDIR}/rootfs/dpkg_partial.log"
deb_dl_dir_export "${WORKDIR}/rootfs" "${distro}"
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH v11 17/21] always create apt-cache dirs in deb_dl_dir_import
2022-05-23 14:35 [PATCH v11 00/21] Sbuild/Schroot migration Anton Mikanovich
` (15 preceding siblings ...)
2022-05-23 14:36 ` [PATCH v11 16/21] fix: support build of packages with epoch version Anton Mikanovich
@ 2022-05-23 14:36 ` Anton Mikanovich
2022-05-23 14:36 ` [PATCH v11 18/21] avoid absolute SCHROOT_* paths to improve caching Anton Mikanovich
` (4 subsequent siblings)
21 siblings, 0 replies; 23+ messages in thread
From: Anton Mikanovich @ 2022-05-23 14:36 UTC (permalink / raw)
To: isar-users; +Cc: Felix Moessbauer
From: Felix Moessbauer <felix.moessbauer@siemens.com>
This patch ensures that the deb_dl_dir_import function
always creates the apt-cache directories under ${PP}/rootfs,
no matter if files are in the download cache or not.
By that, downstream tasks can safely assume that the apt-cache
is created and place files there.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
meta/classes/deb-dl-dir.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index f479b29..188cb83 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -78,8 +78,8 @@ debsrc_download() {
deb_dl_dir_import() {
export pc="${DEBDIR}/${2}"
export rootfs="${1}"
- [ ! -d "${pc}" ] && return 0
sudo mkdir -p "${rootfs}"/var/cache/apt/archives/
+ [ ! -d "${pc}" ] && return 0
flock -s "${pc}".lock -c '
set -e
printenv | grep -q BB_VERBOSE_LOGS && set -x
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH v11 18/21] avoid absolute SCHROOT_* paths to improve caching
2022-05-23 14:35 [PATCH v11 00/21] Sbuild/Schroot migration Anton Mikanovich
` (16 preceding siblings ...)
2022-05-23 14:36 ` [PATCH v11 17/21] always create apt-cache dirs in deb_dl_dir_import Anton Mikanovich
@ 2022-05-23 14:36 ` Anton Mikanovich
2022-05-23 14:36 ` [PATCH v11 19/21] dpkg-base: Cleanup on schroot fail Anton Mikanovich
` (3 subsequent siblings)
21 siblings, 0 replies; 23+ messages in thread
From: Anton Mikanovich @ 2022-05-23 14:36 UTC (permalink / raw)
To: isar-users; +Cc: Felix Moessbauer
From: Felix Moessbauer <felix.moessbauer@siemens.com>
This patch reworks the logic to setup the SCHROOT_* bitbake
variables by making sure nothing is expanded.
By that, no absolute paths into the build host are exposed,
which finally makes the recipe cachable.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
meta/classes/sbuild.bbclass | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index a78b15b..f9bfaaf 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -14,11 +14,11 @@ python __anonymous() {
if mode == "0" or d.getVar('HOST_ARCH') == distro_arch or \
(d.getVar('HOST_DISTRO') == "debian-stretch" and distro_arch == "i386"):
d.setVar('SBUILD_HOST_ARCH', distro_arch)
- d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_TARGET_DIR'))
+ d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_TARGET_DIR', False))
dep = "sbuild-chroot-target:do_build"
else:
d.setVar('SBUILD_HOST_ARCH', d.getVar('HOST_ARCH'))
- d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_HOST_DIR'))
+ d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_HOST_DIR', False))
dep = "sbuild-chroot-host:do_build"
d.setVar('SCHROOT_DEP', dep)
}
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH v11 19/21] dpkg-base: Cleanup on schroot fail
2022-05-23 14:35 [PATCH v11 00/21] Sbuild/Schroot migration Anton Mikanovich
` (17 preceding siblings ...)
2022-05-23 14:36 ` [PATCH v11 18/21] avoid absolute SCHROOT_* paths to improve caching Anton Mikanovich
@ 2022-05-23 14:36 ` Anton Mikanovich
2022-05-23 14:36 ` [PATCH v11 20/21] locate generated binary package in WORKDIR Anton Mikanovich
` (2 subsequent siblings)
21 siblings, 0 replies; 23+ messages in thread
From: Anton Mikanovich @ 2022-05-23 14:36 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
Cleanup schroot configs in case schroot exit with non zero state.
Handle all possible interrupt cases with trap.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
meta/classes/dpkg-base.bbclass | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 7948b23..39ab04f 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -106,6 +106,13 @@ python() {
do_apt_fetch() {
E="${@ isar_export_proxies(d)}"
schroot_create_configs
+
+ schroot_cleanup() {
+ schroot_delete_configs
+ }
+ trap 'exit 1' INT HUP QUIT TERM ALRM USR1
+ trap 'schroot_cleanup' EXIT
+
for uri in "${SRC_APT}"; do
schroot -d / -c ${SBUILD_CHROOT} -- \
sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only-source source "$2"' my_script "${DISTRO}" "${uri}"
@@ -125,6 +132,13 @@ do_apt_fetch[depends] += "${SCHROOT_DEP}"
do_apt_unpack() {
rm -rf ${S}
schroot_create_configs
+
+ schroot_cleanup() {
+ schroot_delete_configs
+ }
+ trap 'exit 1' INT HUP QUIT TERM ALRM USR1
+ trap 'schroot_cleanup' EXIT
+
for uri in "${SRC_APT}"; do
schroot -d / -c ${SBUILD_CHROOT} -- \
sh -c ' \
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH v11 20/21] locate generated binary package in WORKDIR
2022-05-23 14:35 [PATCH v11 00/21] Sbuild/Schroot migration Anton Mikanovich
` (18 preceding siblings ...)
2022-05-23 14:36 ` [PATCH v11 19/21] dpkg-base: Cleanup on schroot fail Anton Mikanovich
@ 2022-05-23 14:36 ` Anton Mikanovich
2022-05-23 14:36 ` [PATCH v11 21/21] sbuild: Add changelog entry Anton Mikanovich
2022-05-23 14:41 ` [PATCH v11 00/21] Sbuild/Schroot migration Anton Mikanovich
21 siblings, 0 replies; 23+ messages in thread
From: Anton Mikanovich @ 2022-05-23 14:36 UTC (permalink / raw)
To: isar-users; +Cc: Felix Moessbauer
From: Felix Moessbauer <felix.moessbauer@siemens.com>
With sbuilder, the generated binary packages are always
emitted into the WORKDIR indepedent of the value of ${S}.
Prior to this fix, this only worked if ${S} was a direct
subfolder of ${WORKDIR} (but no nested folder).
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
meta/classes/dpkg-base.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 39ab04f..79c1ce1 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -296,7 +296,7 @@ do_clean[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
do_deploy_deb() {
deb_clean
repo_add_packages "${REPO_ISAR_DIR}"/"${DISTRO}" \
- "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" ${S}/../*.deb
+ "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" ${WORKDIR}/*.deb
}
addtask deploy_deb after do_dpkg_build before do_build
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH v11 21/21] sbuild: Add changelog entry
2022-05-23 14:35 [PATCH v11 00/21] Sbuild/Schroot migration Anton Mikanovich
` (19 preceding siblings ...)
2022-05-23 14:36 ` [PATCH v11 20/21] locate generated binary package in WORKDIR Anton Mikanovich
@ 2022-05-23 14:36 ` Anton Mikanovich
2022-05-23 14:41 ` [PATCH v11 00/21] Sbuild/Schroot migration Anton Mikanovich
21 siblings, 0 replies; 23+ messages in thread
From: Anton Mikanovich @ 2022-05-23 14:36 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
Migration from buildchroot to sbuild will cause API change.
Add changelog entry which also includes do_install_builddeps task
remove mentioning.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
RECIPE-API-CHANGELOG.md | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index f3b3035..092292b 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -397,3 +397,16 @@ New conversions can be added by defining CONVERSION_CMD_type.
- the conversions appends its own type, e.g. the output file of a conversion `xz`
would be ${IMAGE_FULLNAME}.${type}.xz
- a final chown is appended automatically
+
+### Buildchroot no longer used for package building
+
+Packages are now built with sbuild which takes care of dependency
+installation.
+The task do_install_builddeps has been removed.
+
+The migration to sbuild also means that all changes in the rootfs made during
+package building will not be shared between the build sessions of different
+packages and will be lost after a given build session finishes.
+
+Any package build requirements for the rootfs should be satisfied in the
+Debian way via package dependencies.
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [PATCH v11 00/21] Sbuild/Schroot migration
2022-05-23 14:35 [PATCH v11 00/21] Sbuild/Schroot migration Anton Mikanovich
` (20 preceding siblings ...)
2022-05-23 14:36 ` [PATCH v11 21/21] sbuild: Add changelog entry Anton Mikanovich
@ 2022-05-23 14:41 ` Anton Mikanovich
21 siblings, 0 replies; 23+ messages in thread
From: Anton Mikanovich @ 2022-05-23 14:41 UTC (permalink / raw)
To: isar-users
23.05.2022 17:35, Anton Mikanovich wrote:
> This is a patchset showing how sbuild/schroot tools can be integrated
> into Isar build system.
>
> Base schroot images are created with sbuild-chroot-target or, in case
> of cross-build, sbuild-chroot-host recipes. These images are based on
> isar-bootstrap ones, but include some build-related stuff preinstalled.
>
> To use these changes you need to have sbuild and schroot installed and
> configured. User who runs build should be added to sbuild group.
>
> Currently, sbuild-related stuff is supported in latest kas.
>
> If Gilab CI is used, the following configuration changes are required:
> - external dir for schroot overlay should be set in the config
> /etc/gitlab-runner/config.toml:
>
> volumes = ["/path/to/overlay:/var/lib/schroot/union/overlay"]
>
> The patchset was tested with 'meta-iot2050' and 'xenomai-images'
> downstreams in combination with kas image mentioned above.
>
> Current limitations:
> - parallel building of several packages requires more free space in
> comparison with buildchroot-based architecture. This happens due
> the sbuild architecture which uses some basic layer (common for all
> packages) and per-package separate layer (where builddeps are downloaded
> and installed).
>
> Changes sinse v10:
> - Rebase on current next.
This is just a rebase which also includes single addition patches send
after v10.
No functional changes made since v10.
^ permalink raw reply [flat|nested] 23+ messages in thread