* [PATCH v2 0/4] isar-bootstrap cleanups, fixes and improvements
@ 2021-08-11 19:02 Jan Kiszka
2021-08-11 19:02 ` [PATCH v2 1/4] isar-bootstrap: Remove obsolete and incorrect skipping of rebuild Jan Kiszka
` (4 more replies)
0 siblings, 5 replies; 6+ messages in thread
From: Jan Kiszka @ 2021-08-11 19:02 UTC (permalink / raw)
To: isar-users
These changes finally trigger clean rebuilds when changes
DISTRO_APT_SOURCES and PREFERENCES, or any of the file that those vars
reference. In addition, they clean up unneeded duplications in
isar-bootstrap-host/target.
Patches 1 and 2 are unchanged from v1, just bundled here to make clear
that patches 3 and 4 are based on top.
Jan
Jan Kiszka (4):
isar-bootstrap: Remove obsolete and incorrect skipping of rebuild
isar-bootstrap: Consolidate common host and target bits
isar-bootstrap: Further consolidate host and target
isar-bootstrap: Ensure rebuild on changes in
DISTRO_APT_SOURCES/PREFERENCES files
.../isar-bootstrap/isar-bootstrap-host.bb | 37 +--
.../isar-bootstrap/isar-bootstrap-target.bb | 33 --
.../isar-bootstrap/isar-bootstrap.inc | 311 ++++++++++--------
3 files changed, 167 insertions(+), 214 deletions(-)
--
2.31.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH v2 1/4] isar-bootstrap: Remove obsolete and incorrect skipping of rebuild
2021-08-11 19:02 [PATCH v2 0/4] isar-bootstrap cleanups, fixes and improvements Jan Kiszka
@ 2021-08-11 19:02 ` Jan Kiszka
2021-08-11 19:02 ` [PATCH v2 2/4] isar-bootstrap: Consolidate common host and target bits Jan Kiszka
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Jan Kiszka @ 2021-08-11 19:02 UTC (permalink / raw)
To: isar-users
From: Jan Kiszka <jan.kiszka@siemens.com>
This check dates back to the time isar-bootstrap could have run multiple
times for the same distro-arch and was lock-protected to avoid this.
Long history, locks were removed in 1cae951259b3.
Now this check only prevented proper rebuilding when apt sources or
preferences changed.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
.../isar-bootstrap/isar-bootstrap.inc | 202 +++++++++---------
1 file changed, 100 insertions(+), 102 deletions(-)
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 0edefc5..b858c8c 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -256,125 +256,123 @@ isar_bootstrap() {
fi
E="${@ isar_export_proxies(d)}"
export IS_HOST debootstrap_args E
- if [ ! -e "${DEPLOY_ISAR_BOOTSTRAP}" ]; then
- sudo rm -rf --one-file-system "${ROOTFSDIR}"
- if [ "${IS_HOST}" ];then
- deb_dl_dir_import "${ROOTFSDIR}" "${HOST_DISTRO}"
+
+ sudo rm -rf --one-file-system "${ROOTFSDIR}"
+ if [ "${IS_HOST}" ];then
+ deb_dl_dir_import "${ROOTFSDIR}" "${HOST_DISTRO}"
+ else
+ deb_dl_dir_import "${ROOTFSDIR}" "${DISTRO}"
+ fi
+
+ sudo -E -s <<'EOSUDO'
+ set -e
+ if [ ${IS_HOST} ]; then
+ ${DEBOOTSTRAP} $debootstrap_args \
+ ${@get_distro_components_argument(d, True)} \
+ "${@get_distro_suite(d, True)}" \
+ "${ROOTFSDIR}" \
+ "${@get_distro_source(d, True)}" \
+ ${DISTRO_DEBOOTSTRAP_SCRIPT}
else
- deb_dl_dir_import "${ROOTFSDIR}" "${DISTRO}"
+ ${DEBOOTSTRAP} $debootstrap_args \
+ --arch="${DISTRO_ARCH}" \
+ ${@get_distro_components_argument(d, False)} \
+ "${@get_distro_suite(d, False)}" \
+ "${ROOTFSDIR}" \
+ "${@get_distro_source(d, False)}" \
+ ${DISTRO_DEBOOTSTRAP_SCRIPT}
fi
- sudo -E -s <<'EOSUDO'
- set -e
- if [ ${IS_HOST} ]; then
- ${DEBOOTSTRAP} $debootstrap_args \
- ${@get_distro_components_argument(d, True)} \
- "${@get_distro_suite(d, True)}" \
- "${ROOTFSDIR}" \
- "${@get_distro_source(d, True)}" \
- ${DISTRO_DEBOOTSTRAP_SCRIPT}
- else
- ${DEBOOTSTRAP} $debootstrap_args \
- --arch="${DISTRO_ARCH}" \
- ${@get_distro_components_argument(d, False)} \
- "${@get_distro_suite(d, False)}" \
- "${ROOTFSDIR}" \
- "${@get_distro_source(d, False)}" \
- ${DISTRO_DEBOOTSTRAP_SCRIPT}
+ # Install apt config
+ mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d"
+ install -v -m644 "${APTPREFS}" \
+ "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap"
+ mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
+ line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} main"
+ if [ -z "${BASE_REPO_KEY}" ]; then
+ line="[trusted=yes] ${line}"
fi
+ echo "deb ${line}" > "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
+ echo "deb-src ${line}" >> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
- # Install apt config
- mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d"
- install -v -m644 "${APTPREFS}" \
- "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap"
- mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
- if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
- line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} main"
- if [ -z "${BASE_REPO_KEY}" ]; then
- line="[trusted=yes] ${line}"
- fi
- echo "deb ${line}" > "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
- echo "deb-src ${line}" >> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
-
- mkdir -p ${ROOTFSDIR}/base-apt
- mount --bind ${REPO_BASE_DIR} ${ROOTFSDIR}/base-apt
- else
- install -v -m644 "${APTSRCS}" \
- "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
- fi
- install -v -m644 "${APTSRCS_INIT}" \
- "${ROOTFSDIR}/etc/apt/sources-list"
- rm -f "${ROOTFSDIR}/etc/apt/sources.list"
- rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"*
- mkdir -p "${ROOTFSDIR}/etc/apt/apt.conf.d"
- install -v -m644 "${WORKDIR}/isar-apt.conf" \
- "${ROOTFSDIR}/etc/apt/apt.conf.d/50isar.conf"
- if [ "${@get_distro_needs_gpg_support(d)}" = "gnupg" ]; then
- MY_GPGHOME="$(chroot "${ROOTFSDIR}" mktemp -d /tmp/gpghomeXXXXXXXXXX)"
- echo "Created temporary directory ${MY_GPGHOME} for gpg-agent"
- export GNUPGHOME="${MY_GPGHOME}"
- chroot "${ROOTFSDIR}" gpg-agent --daemon
- APT_KEY_APPEND="--homedir ${MY_GPGHOME}"
- fi
- find ${APT_KEYS_DIR}/ -type f | while read keyfile
- do
- kfn="$(basename $keyfile)"
- cp $keyfile "${ROOTFSDIR}/tmp/$kfn"
- chroot "${ROOTFSDIR}" /usr/bin/apt-key \
- --keyring ${THIRD_PARTY_APT_KEYRING} ${APT_KEY_APPEND} add "/tmp/$kfn"
- rm "${ROOTFSDIR}/tmp/$kfn"
- done
- if [ -d "${MY_GPGHOME}" ]; then
- echo "Killing gpg-agent for ${MY_GPGHOME}"
- chroot "${ROOTFSDIR}" gpgconf --kill gpg-agent && /bin/rm -rf "${MY_GPGHOME}"
- fi
+ mkdir -p ${ROOTFSDIR}/base-apt
+ mount --bind ${REPO_BASE_DIR} ${ROOTFSDIR}/base-apt
+ else
+ install -v -m644 "${APTSRCS}" \
+ "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
+ fi
+ install -v -m644 "${APTSRCS_INIT}" "${ROOTFSDIR}/etc/apt/sources-list"
+ rm -f "${ROOTFSDIR}/etc/apt/sources.list"
+ rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"*
+ mkdir -p "${ROOTFSDIR}/etc/apt/apt.conf.d"
+ install -v -m644 "${WORKDIR}/isar-apt.conf" \
+ "${ROOTFSDIR}/etc/apt/apt.conf.d/50isar.conf"
+ if [ "${@get_distro_needs_gpg_support(d)}" = "gnupg" ]; then
+ MY_GPGHOME="$(chroot "${ROOTFSDIR}" mktemp -d /tmp/gpghomeXXXXXXXXXX)"
+ echo "Created temporary directory ${MY_GPGHOME} for gpg-agent"
+ export GNUPGHOME="${MY_GPGHOME}"
+ chroot "${ROOTFSDIR}" gpg-agent --daemon
+ APT_KEY_APPEND="--homedir ${MY_GPGHOME}"
+ fi
+ find ${APT_KEYS_DIR}/ -type f | while read keyfile
+ do
+ kfn="$(basename $keyfile)"
+ cp $keyfile "${ROOTFSDIR}/tmp/$kfn"
+ chroot "${ROOTFSDIR}" /usr/bin/apt-key \
+ --keyring ${THIRD_PARTY_APT_KEYRING} ${APT_KEY_APPEND} add "/tmp/$kfn"
+ rm "${ROOTFSDIR}/tmp/$kfn"
+ done
+ if [ -d "${MY_GPGHOME}" ]; then
+ echo "Killing gpg-agent for ${MY_GPGHOME}"
+ chroot "${ROOTFSDIR}" gpgconf --kill gpg-agent && /bin/rm -rf "${MY_GPGHOME}"
+ fi
- if [ "${@get_distro_suite(d, True)}" = "stretch" ] && [ "${@get_host_release().split('.')[0]}" -lt "4" ]; then
- install -v -m644 "${WORKDIR}/isar-apt-fallback.conf" \
- "${ROOTFSDIR}/etc/apt/apt.conf.d/55isar-fallback.conf"
- fi
+ if [ "${@get_distro_suite(d, True)}" = "stretch" ] && [ "${@get_host_release().split('.')[0]}" -lt "4" ]; then
+ install -v -m644 "${WORKDIR}/isar-apt-fallback.conf" \
+ "${ROOTFSDIR}/etc/apt/apt.conf.d/55isar-fallback.conf"
+ fi
- # Set locale
- install -v -m644 "${WORKDIR}/locale" "${ROOTFSDIR}/etc/locale"
+ # Set locale
+ install -v -m644 "${WORKDIR}/locale" "${ROOTFSDIR}/etc/locale"
- sed -i '/en_US.UTF-8 UTF-8/s/^#//g' "${ROOTFSDIR}/etc/locale.gen"
- chroot "${ROOTFSDIR}" /usr/sbin/locale-gen
+ sed -i '/en_US.UTF-8 UTF-8/s/^#//g' "${ROOTFSDIR}/etc/locale.gen"
+ chroot "${ROOTFSDIR}" /usr/sbin/locale-gen
- # setup chroot
- install -v -m755 "${WORKDIR}/chroot-setup.sh" "${ROOTFSDIR}/chroot-setup.sh"
- "${ROOTFSDIR}/chroot-setup.sh" "setup" "${ROOTFSDIR}"
+ # setup chroot
+ install -v -m755 "${WORKDIR}/chroot-setup.sh" "${ROOTFSDIR}/chroot-setup.sh"
+ "${ROOTFSDIR}/chroot-setup.sh" "setup" "${ROOTFSDIR}"
- # update APT
- mount --rbind /dev ${ROOTFSDIR}/dev
- mount --make-rslave ${ROOTFSDIR}/dev
- mount -t proc none ${ROOTFSDIR}/proc
- mount --rbind /sys ${ROOTFSDIR}/sys
- mount --make-rslave ${ROOTFSDIR}/sys
+ # update APT
+ mount --rbind /dev ${ROOTFSDIR}/dev
+ mount --make-rslave ${ROOTFSDIR}/dev
+ mount -t proc none ${ROOTFSDIR}/proc
+ mount --rbind /sys ${ROOTFSDIR}/sys
+ mount --make-rslave ${ROOTFSDIR}/sys
- export DEBIAN_FRONTEND=noninteractive
+ export DEBIAN_FRONTEND=noninteractive
- if [ ${IS_HOST} ]; then
- chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${DISTRO_ARCH}
- fi
+ if [ ${IS_HOST} ]; then
+ chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${DISTRO_ARCH}
+ fi
- if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then
- chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${COMPAT_DISTRO_ARCH}
- fi
+ if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then
+ chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${COMPAT_DISTRO_ARCH}
+ fi
- chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y
- chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f
- chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \
- -o Debug::pkgProblemResolver=yes
+ chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y
+ chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f
+ chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \
+ -o Debug::pkgProblemResolver=yes
- umount -l "${ROOTFSDIR}/dev"
- umount -l "${ROOTFSDIR}/proc"
- umount -l "${ROOTFSDIR}/sys"
- umount -l "${ROOTFSDIR}/base-apt" || true
+ umount -l "${ROOTFSDIR}/dev"
+ umount -l "${ROOTFSDIR}/proc"
+ umount -l "${ROOTFSDIR}/sys"
+ umount -l "${ROOTFSDIR}/base-apt" || true
- # Finalize debootstrap by setting the link in deploy
- ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
+ # Finalize debootstrap by setting the link in deploy
+ ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
EOSUDO
- fi
if [ "${IS_HOST}" ];then
deb_dl_dir_export "${ROOTFSDIR}" "${HOST_DISTRO}"
else
--
2.31.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH v2 2/4] isar-bootstrap: Consolidate common host and target bits
2021-08-11 19:02 [PATCH v2 0/4] isar-bootstrap cleanups, fixes and improvements Jan Kiszka
2021-08-11 19:02 ` [PATCH v2 1/4] isar-bootstrap: Remove obsolete and incorrect skipping of rebuild Jan Kiszka
@ 2021-08-11 19:02 ` Jan Kiszka
2021-08-11 19:02 ` [PATCH v2 3/4] isar-bootstrap: Further consolidate host and target Jan Kiszka
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Jan Kiszka @ 2021-08-11 19:02 UTC (permalink / raw)
To: isar-users
From: Jan Kiszka <jan.kiszka@siemens.com>
Avoids duplicating too much identical logic.
The key element is introducing DISTRO_VARS_PREFIX that adds "HOST_" for
the buildchroot-host case.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
.../isar-bootstrap/isar-bootstrap-host.bb | 31 +---------------
.../isar-bootstrap/isar-bootstrap-target.bb | 27 --------------
.../isar-bootstrap/isar-bootstrap.inc | 37 ++++++++++++++++++-
3 files changed, 38 insertions(+), 57 deletions(-)
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
index c985383..8f1fced 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
@@ -9,42 +9,15 @@ Description = "Minimal host Debian root file system"
DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-host_${DISTRO}-${DISTRO_ARCH}"
+DISTRO_VARS_PREFIX = "HOST_"
+
require isar-bootstrap.inc
HOST_DISTRO_BOOTSTRAP_KEYS ?= ""
DISTRO_BOOTSTRAP_KEYS = "${HOST_DISTRO_BOOTSTRAP_KEYS}"
-do_apt_config_prepare[dirs] = "${WORKDIR}"
-do_apt_config_prepare[vardeps] += "\
- APTPREFS \
- HOST_DISTRO_APT_PREFERENCES \
- DEBDISTRONAME \
- APTSRCS \
- HOST_DISTRO_APT_SOURCES \
- DEPLOY_ISAR_BOOTSTRAP \
- "
-python do_apt_config_prepare() {
- apt_preferences_out = d.getVar("APTPREFS", True)
- apt_preferences_list = (
- d.getVar("HOST_DISTRO_APT_PREFERENCES", True) or ""
- ).split()
- aggregate_files(d, apt_preferences_list, apt_preferences_out)
-
- apt_sources_out = d.getVar("APTSRCS", True)
- apt_sources_init_out = d.getVar("APTSRCS_INIT", True)
- apt_sources_list = (
- d.getVar("HOST_DISTRO_APT_SOURCES", True) or ""
- ).split()
-
- aggregate_files(d, apt_sources_list, apt_sources_init_out)
- aggregate_aptsources_list(d, apt_sources_list, apt_sources_out)
-}
-addtask apt_config_prepare before do_bootstrap after do_unpack
-
OVERRIDES_append = ":${@get_distro_needs_https_support(d, True)}"
-do_bootstrap[vardeps] += "HOST_DISTRO_APT_SOURCES"
do_bootstrap() {
isar_bootstrap --host
}
-addtask bootstrap before do_build after do_generate_keyrings
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
index 80e7f40..7158a86 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
@@ -11,35 +11,8 @@ DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${DISTRO}-${DISTRO_ARCH}"
require isar-bootstrap.inc
-do_apt_config_prepare[dirs] = "${WORKDIR}"
-do_apt_config_prepare[vardeps] += "\
- APTPREFS \
- DISTRO_APT_PREFERENCES \
- DEBDISTRONAME \
- APTSRCS \
- DISTRO_APT_SOURCES \
- DEPLOY_ISAR_BOOTSTRAP \
- "
-python do_apt_config_prepare() {
- apt_preferences_out = d.getVar("APTPREFS", True)
- apt_preferences_list = (
- d.getVar("DISTRO_APT_PREFERENCES", True) or ""
- ).split()
- aggregate_files(d, apt_preferences_list, apt_preferences_out)
-
- apt_sources_out = d.getVar("APTSRCS", True)
- apt_sources_init_out = d.getVar("APTSRCS_INIT", True)
- apt_sources_list = (d.getVar("DISTRO_APT_SOURCES", True) or "").split()
-
- aggregate_files(d, apt_sources_list, apt_sources_init_out)
- aggregate_aptsources_list(d, apt_sources_list, apt_sources_out)
-}
-addtask apt_config_prepare before do_bootstrap after do_unpack
-
OVERRIDES_append = ":${@get_distro_needs_https_support(d, False)}"
-do_bootstrap[vardeps] += "DISTRO_APT_SOURCES"
do_bootstrap() {
isar_bootstrap
}
-addtask bootstrap before do_build after do_generate_keyrings
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index b858c8c..a846389 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -27,6 +27,8 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"
DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg"
DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = "${@https_support(d)}"
+DISTRO_VARS_PREFIX ?= ""
+
inherit deb-dl-dir
python () {
@@ -220,15 +222,48 @@ do_generate_keyrings() {
}
addtask generate_keyrings before do_build after do_unpack
+do_apt_config_prepare[dirs] = "${WORKDIR}"
+do_apt_config_prepare[vardeps] += " \
+ APTPREFS \
+ ${DISTRO_VARS_PREFIX}DISTRO_APT_PREFERENCES \
+ DEBDISTRONAME \
+ APTSRCS \
+ ${DISTRO_VARS_PREFIX}DISTRO_APT_SOURCES \
+ DEPLOY_ISAR_BOOTSTRAP \
+ "
+python do_apt_config_prepare() {
+ apt_preferences_out = d.getVar("APTPREFS", True)
+ apt_preferences_list = (
+ d.getVar(d.getVar("DISTRO_VARS_PREFIX") + "DISTRO_APT_PREFERENCES", True) or ""
+ ).split()
+ aggregate_files(d, apt_preferences_list, apt_preferences_out)
+
+ apt_sources_out = d.getVar("APTSRCS", True)
+ apt_sources_init_out = d.getVar("APTSRCS_INIT", True)
+ apt_sources_list = (
+ d.getVar(d.getVar("DISTRO_VARS_PREFIX") + "DISTRO_APT_SOURCES", True) or ""
+ ).split()
+
+ aggregate_files(d, apt_sources_list, apt_sources_init_out)
+ aggregate_aptsources_list(d, apt_sources_list, apt_sources_out)
+}
+addtask apt_config_prepare before do_bootstrap after do_unpack
+
def get_host_release():
import platform
rel = platform.release()
return rel
-do_bootstrap[vardeps] += "DISTRO_APT_PREMIRRORS ISAR_ENABLE_COMPAT_ARCH"
+do_bootstrap[vardeps] += " \
+ DISTRO_APT_PREMIRRORS \
+ ISAR_ENABLE_COMPAT_ARCH \
+ ${DISTRO_VARS_PREFIX}DISTRO_APT_SOURCES \
+ "
do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP}"
do_bootstrap[depends] = "base-apt:do_cache isar-apt:do_cache_config"
+addtask bootstrap before do_build after do_generate_keyrings
+
isar_bootstrap() {
IS_HOST=""
while true; do
--
2.31.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH v2 3/4] isar-bootstrap: Further consolidate host and target
2021-08-11 19:02 [PATCH v2 0/4] isar-bootstrap cleanups, fixes and improvements Jan Kiszka
2021-08-11 19:02 ` [PATCH v2 1/4] isar-bootstrap: Remove obsolete and incorrect skipping of rebuild Jan Kiszka
2021-08-11 19:02 ` [PATCH v2 2/4] isar-bootstrap: Consolidate common host and target bits Jan Kiszka
@ 2021-08-11 19:02 ` Jan Kiszka
2021-08-11 19:02 ` [PATCH v2 4/4] isar-bootstrap: Ensure rebuild on changes in DISTRO_APT_SOURCES/PREFERENCES files Jan Kiszka
2021-08-24 13:48 ` [PATCH v2 0/4] isar-bootstrap cleanups, fixes and improvements Anton Mikanovich
4 siblings, 0 replies; 6+ messages in thread
From: Jan Kiszka @ 2021-08-11 19:02 UTC (permalink / raw)
To: isar-users
From: Jan Kiszka <jan.kiszka@siemens.com>
Control the mode via the boolean var BOOTSTRAP_FOR_HOST, deriving
DISTRO_VARS_PREFIX from that. Add BOOTSTRAP_DISTRO which resolves to the
content of DISTRO or HOST_DISTRO, according to BOOTSTRAP_FOR_HOST.
Furthermore, use DISTRO_VARS_PREFIX in get_aptsources_list, rather than
passing an is_host parameter down the call chain. There are two cases
where is_host has so far hard-coded:
- "${@get_distro_suite(d, True)}-${COMPAT_DISTRO_ARCH}" = "stretch-i386"
- "${@get_distro_suite(d, True)}" = "stretch" && kernel-major-version < 4"
In both cases, it was at least inaccurate to hard-code is_host=True so
far. We now use the actual distro that is to be bootstrapped.
As the --host parameter of isar_bootstrap is now obsolete, we can fold
that function into a common do_bootstrap.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
.../isar-bootstrap/isar-bootstrap-host.bb | 8 +-
.../isar-bootstrap/isar-bootstrap-target.bb | 6 -
.../isar-bootstrap/isar-bootstrap.inc | 112 +++++++-----------
3 files changed, 46 insertions(+), 80 deletions(-)
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
index 8f1fced..4f90fd0 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
@@ -9,15 +9,9 @@ Description = "Minimal host Debian root file system"
DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-host_${DISTRO}-${DISTRO_ARCH}"
-DISTRO_VARS_PREFIX = "HOST_"
+BOOTSTRAP_FOR_HOST = "1"
require isar-bootstrap.inc
HOST_DISTRO_BOOTSTRAP_KEYS ?= ""
DISTRO_BOOTSTRAP_KEYS = "${HOST_DISTRO_BOOTSTRAP_KEYS}"
-
-OVERRIDES_append = ":${@get_distro_needs_https_support(d, True)}"
-
-do_bootstrap() {
- isar_bootstrap --host
-}
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
index 7158a86..c66cb3b 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
@@ -10,9 +10,3 @@ Description = "Minimal target Debian root file system"
DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${DISTRO}-${DISTRO_ARCH}"
require isar-bootstrap.inc
-
-OVERRIDES_append = ":${@get_distro_needs_https_support(d, False)}"
-
-do_bootstrap() {
- isar_bootstrap
-}
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index a846389..f1299f9 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -1,7 +1,7 @@
# Minimal debian root file system
#
# This software is a part of ISAR.
-# Copyright (c) Siemens AG, 2018
+# Copyright (c) Siemens AG, 2018-2021
#
# SPDX-License-Identifier: MIT
@@ -15,6 +15,8 @@ SRC_URI = " \
file://chroot-setup.sh"
PV = "1.0"
+BOOTSTRAP_FOR_HOST ?= "0"
+
DEBOOTSTRAP ?= "qemu-debootstrap"
ROOTFSDIR = "${WORKDIR}/rootfs"
APTPREFS = "${WORKDIR}/apt-preferences"
@@ -26,8 +28,8 @@ DEPLOY_ISAR_BOOTSTRAP ?= ""
DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"
DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg"
DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = "${@https_support(d)}"
-
-DISTRO_VARS_PREFIX ?= ""
+DISTRO_VARS_PREFIX ?= "${@'HOST_' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}"
+BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO')}"
inherit deb-dl-dir
@@ -137,15 +139,12 @@ def aggregate_aptsources_list(d, file_list, file_out):
out_fd.write("\n".encode())
out_fd.write("\n".encode())
-def get_aptsources_list(d, is_host=False):
- if is_host:
- apt_sources_list = (d.getVar("HOST_DISTRO_APT_SOURCES", True) or "").split()
- else:
- apt_sources_list = (d.getVar("DISTRO_APT_SOURCES", True) or "").split()
- return apt_sources_list
+def get_aptsources_list(d):
+ apt_sources_var = d.getVar("DISTRO_VARS_PREFIX") + "DISTRO_APT_SOURCES"
+ return (d.getVar(apt_sources_var, True) or "").split()
-def generate_distro_sources(d, is_host=False):
- apt_sources_list = get_aptsources_list(d, is_host)
+def generate_distro_sources(d):
+ apt_sources_list = get_aptsources_list(d)
for entry in apt_sources_list:
entry_real = bb.parse.resolve_file(entry, d)
with open(entry_real, "r") as in_fd:
@@ -155,28 +154,30 @@ def generate_distro_sources(d, is_host=False):
parsed = get_apt_source_mirror(d, parsed)
yield parsed
-def get_distro_primary_source_entry(d, is_host=False):
- apt_sources_list = get_aptsources_list(d, is_host)
- for source in generate_distro_sources(d, is_host):
+def get_distro_primary_source_entry(d):
+ apt_sources_list = get_aptsources_list(d)
+ for source in generate_distro_sources(d):
if source[0] == "deb":
return source[2:]
return ["", "", ""]
-def get_distro_have_https_source(d, is_host=False):
- return any(source[2].startswith("https://") for source in generate_distro_sources(d, is_host))
+def get_distro_have_https_source(d):
+ return any(source[2].startswith("https://") for source in generate_distro_sources(d))
-def https_support(d, is_host=False):
- if get_distro_suite(d, is_host) == "stretch":
+def https_support(d):
+ if get_distro_suite(d) == "stretch":
return ",apt-transport-https,ca-certificates"
else:
return ",ca-certificates"
-def get_distro_needs_https_support(d, is_host=False):
- if get_distro_have_https_source(d, is_host):
+def get_distro_needs_https_support(d):
+ if get_distro_have_https_source(d):
return "https-support"
else:
return ""
+OVERRIDES_append = ":${@get_distro_needs_https_support(d)}"
+
def get_distro_needs_gpg_support(d):
apt_keys = d.getVar("DISTRO_BOOTSTRAP_KEYS") or ""
apt_keys += " " + (d.getVar("THIRD_PARTY_APT_KEYS") or "")
@@ -187,14 +188,14 @@ def get_distro_needs_gpg_support(d):
OVERRIDES_append = ":${@get_distro_needs_gpg_support(d)}"
-def get_distro_source(d, is_host):
- return get_distro_primary_source_entry(d, is_host)[0]
+def get_distro_source(d):
+ return get_distro_primary_source_entry(d)[0]
-def get_distro_suite(d, is_host):
- return get_distro_primary_source_entry(d, is_host)[1]
+def get_distro_suite(d):
+ return get_distro_primary_source_entry(d)[1]
-def get_distro_components_argument(d, is_host):
- components = get_distro_primary_source_entry(d, is_host)[2]
+def get_distro_components_argument(d):
+ components = get_distro_primary_source_entry(d)[2]
if components and components.strip():
return "--components=" + ",".join(components.split())
else:
@@ -262,23 +263,12 @@ do_bootstrap[vardeps] += " \
do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP}"
do_bootstrap[depends] = "base-apt:do_cache isar-apt:do_cache_config"
-addtask bootstrap before do_build after do_generate_keyrings
-
-isar_bootstrap() {
- IS_HOST=""
- while true; do
- case "$1" in
- --host) IS_HOST=1 ;;
- -*) bbfatal "$0: invalid option specified: $1" ;;
- *) break ;;
- esac
- shift
- done
+do_bootstrap() {
if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then
if [ -z "${COMPAT_DISTRO_ARCH}" ]; then
bbfatal "${DISTRO_ARCH} does not have a compat arch"
fi
- if [ "${@get_distro_suite(d, True)}-${COMPAT_DISTRO_ARCH}" = "stretch-i386" ]; then
+ if [ "${@get_distro_suite(d)}-${COMPAT_DISTRO_ARCH}" = "stretch-i386" ]; then
bbfatal "compat arch build for stretch-i386 not supported"
fi
fi
@@ -290,33 +280,23 @@ isar_bootstrap() {
debootstrap_args="$debootstrap_args --no-check-gpg"
fi
E="${@ isar_export_proxies(d)}"
- export IS_HOST debootstrap_args E
+ export BOOTSTRAP_FOR_HOST debootstrap_args E
sudo rm -rf --one-file-system "${ROOTFSDIR}"
- if [ "${IS_HOST}" ];then
- deb_dl_dir_import "${ROOTFSDIR}" "${HOST_DISTRO}"
- else
- deb_dl_dir_import "${ROOTFSDIR}" "${DISTRO}"
- fi
+ deb_dl_dir_import "${ROOTFSDIR}" "${BOOTSTRAP_DISTRO}"
sudo -E -s <<'EOSUDO'
set -e
- if [ ${IS_HOST} ]; then
- ${DEBOOTSTRAP} $debootstrap_args \
- ${@get_distro_components_argument(d, True)} \
- "${@get_distro_suite(d, True)}" \
- "${ROOTFSDIR}" \
- "${@get_distro_source(d, True)}" \
- ${DISTRO_DEBOOTSTRAP_SCRIPT}
- else
- ${DEBOOTSTRAP} $debootstrap_args \
- --arch="${DISTRO_ARCH}" \
- ${@get_distro_components_argument(d, False)} \
- "${@get_distro_suite(d, False)}" \
- "${ROOTFSDIR}" \
- "${@get_distro_source(d, False)}" \
- ${DISTRO_DEBOOTSTRAP_SCRIPT}
+ if [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then
+ arch_param="--arch=${DISTRO_ARCH}"
fi
+ ${DEBOOTSTRAP} $debootstrap_args \
+ $arch_param \
+ ${@get_distro_components_argument(d)} \
+ "${@get_distro_suite(d)}" \
+ "${ROOTFSDIR}" \
+ "${@get_distro_source(d)}" \
+ ${DISTRO_DEBOOTSTRAP_SCRIPT}
# Install apt config
mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d"
@@ -363,7 +343,7 @@ isar_bootstrap() {
chroot "${ROOTFSDIR}" gpgconf --kill gpg-agent && /bin/rm -rf "${MY_GPGHOME}"
fi
- if [ "${@get_distro_suite(d, True)}" = "stretch" ] && [ "${@get_host_release().split('.')[0]}" -lt "4" ]; then
+ if [ "${@get_distro_suite(d)}" = "stretch" ] && [ "${@get_host_release().split('.')[0]}" -lt "4" ]; then
install -v -m644 "${WORKDIR}/isar-apt-fallback.conf" \
"${ROOTFSDIR}/etc/apt/apt.conf.d/55isar-fallback.conf"
fi
@@ -387,7 +367,7 @@ isar_bootstrap() {
export DEBIAN_FRONTEND=noninteractive
- if [ ${IS_HOST} ]; then
+ if [ "${BOOTSTRAP_FOR_HOST}" = "1" ]; then
chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${DISTRO_ARCH}
fi
@@ -408,13 +388,11 @@ isar_bootstrap() {
# Finalize debootstrap by setting the link in deploy
ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
EOSUDO
- if [ "${IS_HOST}" ];then
- deb_dl_dir_export "${ROOTFSDIR}" "${HOST_DISTRO}"
- else
- deb_dl_dir_export "${ROOTFSDIR}" "${DISTRO}"
- fi
+ deb_dl_dir_export "${ROOTFSDIR}" "${BOOTSTRAP_DISTRO}"
}
+addtask bootstrap before do_build after do_generate_keyrings
+
CLEANFUNCS = "clean_deploy"
clean_deploy() {
rm -f "${DEPLOY_ISAR_BOOTSTRAP}"
--
2.31.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH v2 4/4] isar-bootstrap: Ensure rebuild on changes in DISTRO_APT_SOURCES/PREFERENCES files
2021-08-11 19:02 [PATCH v2 0/4] isar-bootstrap cleanups, fixes and improvements Jan Kiszka
` (2 preceding siblings ...)
2021-08-11 19:02 ` [PATCH v2 3/4] isar-bootstrap: Further consolidate host and target Jan Kiszka
@ 2021-08-11 19:02 ` Jan Kiszka
2021-08-24 13:48 ` [PATCH v2 0/4] isar-bootstrap cleanups, fixes and improvements Anton Mikanovich
4 siblings, 0 replies; 6+ messages in thread
From: Jan Kiszka @ 2021-08-11 19:02 UTC (permalink / raw)
To: isar-users
From: Jan Kiszka <jan.kiszka@siemens.com>
For that purpose, simply add them to SRC_URI because the content of
files there affects the task hash.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index f1299f9..c65f2cb 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -58,6 +58,14 @@ python () {
fetcher = bb.fetch2.Fetch([key], d)
filename = fetcher.localpath(key)
d.appendVar("THIRD_PARTY_APT_KEYFILES", " %s" % filename)
+
+ distro_apt_sources = d.getVar(d.getVar("DISTRO_VARS_PREFIX") + "DISTRO_APT_SOURCES", True)
+ for file in distro_apt_sources.split():
+ d.appendVar("SRC_URI", " file://%s" % bb.parse.resolve_file(file, d))
+
+ distro_apt_preferences = d.getVar(d.getVar("DISTRO_VARS_PREFIX") + "DISTRO_APT_PREFERENCES", True)
+ for file in distro_apt_sources.split():
+ d.appendVar("SRC_URI", " file://%s" % bb.parse.resolve_file(file, d))
}
def aggregate_files(d, file_list, file_out):
--
2.31.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v2 0/4] isar-bootstrap cleanups, fixes and improvements
2021-08-11 19:02 [PATCH v2 0/4] isar-bootstrap cleanups, fixes and improvements Jan Kiszka
` (3 preceding siblings ...)
2021-08-11 19:02 ` [PATCH v2 4/4] isar-bootstrap: Ensure rebuild on changes in DISTRO_APT_SOURCES/PREFERENCES files Jan Kiszka
@ 2021-08-24 13:48 ` Anton Mikanovich
4 siblings, 0 replies; 6+ messages in thread
From: Anton Mikanovich @ 2021-08-24 13:48 UTC (permalink / raw)
To: Jan Kiszka, isar-users
11.08.2021 22:02, Jan Kiszka wrote:
> These changes finally trigger clean rebuilds when changes
> DISTRO_APT_SOURCES and PREFERENCES, or any of the file that those vars
> reference. In addition, they clean up unneeded duplications in
> isar-bootstrap-host/target.
>
> Patches 1 and 2 are unchanged from v1, just bundled here to make clear
> that patches 3 and 4 are based on top.
>
> Jan
Applied to next, thanks.
--
Anton Mikanovich
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-08-24 13:48 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-08-11 19:02 [PATCH v2 0/4] isar-bootstrap cleanups, fixes and improvements Jan Kiszka
2021-08-11 19:02 ` [PATCH v2 1/4] isar-bootstrap: Remove obsolete and incorrect skipping of rebuild Jan Kiszka
2021-08-11 19:02 ` [PATCH v2 2/4] isar-bootstrap: Consolidate common host and target bits Jan Kiszka
2021-08-11 19:02 ` [PATCH v2 3/4] isar-bootstrap: Further consolidate host and target Jan Kiszka
2021-08-11 19:02 ` [PATCH v2 4/4] isar-bootstrap: Ensure rebuild on changes in DISTRO_APT_SOURCES/PREFERENCES files Jan Kiszka
2021-08-24 13:48 ` [PATCH v2 0/4] isar-bootstrap cleanups, fixes and improvements Anton Mikanovich
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox