* [PATCH 0/6] pre-processing pipeline and transient package replacement
@ 2019-04-18 11:25 claudius.heine.ext
2019-04-18 11:25 ` [PATCH 1/6] split up isar-bootstrap helper and implement pre-process pipeline claudius.heine.ext
` (6 more replies)
0 siblings, 7 replies; 14+ messages in thread
From: claudius.heine.ext @ 2019-04-18 11:25 UTC (permalink / raw)
To: isar-users; +Cc: Claudius Heine
From: Claudius Heine <ch@denx.de>
Hi,
here is the tested, documented and slightly modified version of the previous
RFC patch set.
The CI has shown that the RFC patchset was not rebuildable. This
was patched by consolidating the `do_rootfs_prepare`, `do_roofs_configure`
and `do_rootfs_install` into just one task `do_rootfs_install` that now calls
the functions of both previous tasks as well as the `rootfs_prepare` function
that previously was its own task `do_rootfs_prepare`.
Otherwise I added documentation and more verbose commit messages.
regards,
Claudius
Claudius Heine (6):
split up isar-bootstrap helper and implement pre-process pipeline
meta: remove transient package support
meta/classes: add image-locales-extension class
meta/classes: add image-account-extension class
doc: update description of image customization
doc: some fixes
doc/user_manual.md | 52 +++-
meta/classes/image-account-extension.bbclass | 253 ++++++++++++++++++
meta/classes/image-locales-extension.bbclass | 97 +++++++
meta/classes/image.bbclass | 30 +--
meta/classes/isar-bootstrap-helper.bbclass | 143 ----------
meta/classes/rootfs.bbclass | 176 +++++++++++-
.../isar-bootstrap/isar-bootstrap.inc | 5 +
.../buildchroot/buildchroot-host.bb | 16 +-
.../buildchroot/buildchroot-target.bb | 2 -
.../buildchroot/buildchroot.inc | 26 +-
meta/recipes-devtools/sdkchroot/sdkchroot.bb | 47 ++--
.../isar-cfg-localepurge/files/postinst | 15 --
.../isar-cfg-localepurge.bb | 70 -----
.../isar-cfg-rootpw/files/postinst.tmpl | 19 --
.../isar-cfg-rootpw/isar-cfg-rootpw.bb | 19 --
15 files changed, 624 insertions(+), 346 deletions(-)
create mode 100644 meta/classes/image-account-extension.bbclass
create mode 100644 meta/classes/image-locales-extension.bbclass
delete mode 100644 meta/classes/isar-bootstrap-helper.bbclass
delete mode 100644 meta/recipes-support/isar-cfg-localepurge/files/postinst
delete mode 100644 meta/recipes-support/isar-cfg-localepurge/isar-cfg-localepurge.bb
delete mode 100644 meta/recipes-support/isar-cfg-rootpw/files/postinst.tmpl
delete mode 100644 meta/recipes-support/isar-cfg-rootpw/isar-cfg-rootpw.bb
--
2.20.1
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH 1/6] split up isar-bootstrap helper and implement pre-process pipeline
2019-04-18 11:25 [PATCH 0/6] pre-processing pipeline and transient package replacement claudius.heine.ext
@ 2019-04-18 11:25 ` claudius.heine.ext
2019-04-18 11:25 ` [PATCH 2/6] meta: remove transient package support claudius.heine.ext
` (5 subsequent siblings)
6 siblings, 0 replies; 14+ messages in thread
From: claudius.heine.ext @ 2019-04-18 11:25 UTC (permalink / raw)
To: isar-users; +Cc: Claudius Heine
From: Claudius Heine <ch@denx.de>
This removes the isar-bootstrap-helper.bbclass and integrates its
functionality into the rootfs.bbclass.
It introduces the rootfs_install task that runs through multiple
functions from command lists in order to allow easy integration of
custom functions.
The introduced functions are, in general order of execution:
rootfs_prepare: copy the isar-bootstrap rootfs base to workdir
rootfs_configure_*: basic configuration of the root file system,
before any custom packages where installed. (`/dev`, `/proc`, etc.
are not mounted). Functions are added to `ROOTFS_CONFIGURE_COMMAND`.
rootfs_do_mount: mount standard directories and repositories into the
rootfs
do_rootfs_install_*: downloading and installing packages. Functions
are added to `ROOTFS_INSTALL_COMMAND`.
Signed-off-by: Claudius Heine <ch@denx.de>
---
meta/classes/image.bbclass | 24 +--
meta/classes/isar-bootstrap-helper.bbclass | 143 ---------------
meta/classes/rootfs.bbclass | 172 +++++++++++++++++-
.../isar-bootstrap/isar-bootstrap.inc | 5 +
.../buildchroot/buildchroot-host.bb | 16 +-
.../buildchroot/buildchroot-target.bb | 2 -
.../buildchroot/buildchroot.inc | 26 ++-
meta/recipes-devtools/sdkchroot/sdkchroot.bb | 47 +++--
8 files changed, 227 insertions(+), 208 deletions(-)
delete mode 100644 meta/classes/isar-bootstrap-helper.bbclass
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 6881d37..c795414 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -53,8 +53,10 @@ image_do_mounts() {
buildchroot_do_mounts
}
-inherit isar-bootstrap-helper
-ROOTFS_FEATURES += "finalize-rootfs"
+ROOTFSDIR = "${IMAGE_ROOTFS}"
+ROOTFS_FEATURES += "clean-package-cache finalize-rootfs"
+ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}"
+
inherit rootfs
inherit image-sdk-extension
inherit image-cache-extension
@@ -121,8 +123,10 @@ python set_image_size () {
d.setVarFlag('ROOTFS_SIZE', 'export', '1')
}
-do_image_gen_fstab() {
- cat > ${WORKDIR}/fstab << EOF
+ROOTFS_CONFIGURE_COMMAND += "image_configure_fstab"
+image_configure_fstab[weight] = "2"
+image_configure_fstab() {
+ sudo tee '${IMAGE_ROOTFS}/etc/fstab' << EOF
# Begin /etc/fstab
/dev/root / auto defaults 0 0
proc /proc proc nosuid,noexec,nodev 0 0
@@ -134,18 +138,6 @@ devtmpfs /dev devtmpfs mode=0755,nosuid 0 0
# End /etc/fstab
EOF
}
-addtask image_gen_fstab before do_rootfs_install
-
-do_rootfs_install[depends] = "isar-apt:do_cache_config isar-bootstrap-target:do_bootstrap"
-do_rootfs_install[deptask] = "do_deploy_deb"
-do_rootfs_install[root_cleandirs] = "${IMAGE_ROOTFS} \
- ${IMAGE_ROOTFS}/isar-apt"
-do_rootfs_install() {
- setup_root_file_system --clean --keep-apt-cache \
- --fstab "${WORKDIR}/fstab" \
- "${IMAGE_ROOTFS}" ${IMAGE_PREINSTALL} ${IMAGE_INSTALL}
-}
-addtask rootfs_install before do_build after do_unpack
do_copy_boot_files[dirs] = "${DEPLOY_DIR_IMAGE}"
do_copy_boot_files() {
diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
deleted file mode 100644
index 0d77929..0000000
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ /dev/null
@@ -1,143 +0,0 @@
-# Helper functions for using isar-bootstrap
-#
-# This software is a part of ISAR.
-# Copyright (c) Siemens AG, 2018
-#
-# SPDX-License-Identifier: MIT
-
-IMAGE_TRANSIENT_PACKAGES ??= ""
-
-def reverse_bb_array(d, varname):
- array = d.getVar(varname, True)
- if array is None:
- return None
- array = reversed(array.split())
- return " ".join(i for i in array)
-
-
-update_etc_os_release() {
- OS_RELEASE_BUILD_ID=""
- OS_RELEASE_VARIANT=""
- while true; do
- case "$1" in
- --build-id) OS_RELEASE_BUILD_ID=$2; shift ;;
- --variant) OS_RELEASE_VARIANT=$2; shift ;;
- -*) bbfatal "$0: invalid option specified: $1" ;;
- *) break ;;
- esac
- shift
- done
- ROOTFSDIR="$1"
-
- if [ -n "${OS_RELEASE_BUILD_ID}" ]; then
- sudo sed -i '/^BUILD_ID=.*/d' ${ROOTFSDIR}/etc/os-release
- echo "BUILD_ID=\"${OS_RELEASE_BUILD_ID}\"" | \
- sudo tee -a ${ROOTFSDIR}/etc/os-release
- fi
- if [ -n "${OS_RELEASE_VARIANT}" ]; then
- sudo sed -i '/^VARIANT=.*/d' ${ROOTFSDIR}/etc/os-release
- echo "VARIANT=\"${OS_RELEASE_VARIANT}\"" | \
- sudo tee -a ${ROOTFSDIR}/etc/os-release
- fi
-}
-
-setup_root_file_system() {
- CLEAN=""
- COPYISARAPT=""
- FSTAB=""
- ROOTFS_ARCH="${DISTRO_ARCH}"
- ROOTFS_DISTRO="${DISTRO}"
- while true; do
- case "$1" in
- --clean) CLEAN=1 ;;
- --copyisarapt) COPYISARAPT=1 ;;
- --fstab) FSTAB=$2; shift ;;
- --host-arch) ROOTFS_ARCH=${HOST_ARCH} ;;
- --host-distro) ROOTFS_DISTRO=${HOST_DISTRO} ;;
- --keep-apt-cache) KEEP_APT_CACHE=1 ;;
- -*) bbfatal "$0: invalid option specified: $1" ;;
- *) break ;;
- esac
- shift
- done
- ROOTFSDIR="$1"
- shift
- PACKAGES="$@"
- APT_ARGS="install --yes -o Debug::pkgProblemResolver=yes"
- CLEAN_FILES="${ROOTFSDIR}/etc/hostname ${ROOTFSDIR}/etc/resolv.conf"
-
- sudo cp -Trpfx \
- "${DEPLOY_DIR_BOOTSTRAP}/$ROOTFS_DISTRO-$ROOTFS_ARCH/" \
- "$ROOTFSDIR"
- [ -n "${FSTAB}" ] && cat ${FSTAB} | sudo tee "$ROOTFSDIR/etc/fstab"
-
- echo "deb [trusted=yes] file:///isar-apt ${DEBDISTRONAME} main" | \
- sudo tee "$ROOTFSDIR/etc/apt/sources.list.d/isar-apt.list" >/dev/null
-
- echo "Package: *\nPin: release n=${DEBDISTRONAME}\nPin-Priority: 1000" | \
- sudo tee "$ROOTFSDIR/etc/apt/preferences.d/isar" >/dev/null
-
- if [ ${COPYISARAPT} ]; then
- sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} $ROOTFSDIR/isar-apt
- else
- sudo mount --bind ${REPO_ISAR_DIR}/${DISTRO} $ROOTFSDIR/isar-apt
- fi
-
- if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
- sudo mount --bind ${REPO_BASE_DIR} ${ROOTFSDIR}/base-apt
- fi
-
- sudo mount --rbind /dev ${ROOTFSDIR}/dev
- sudo mount --make-rslave ${ROOTFSDIR}/dev
- sudo mount -t proc none $ROOTFSDIR/proc
- sudo mount --rbind /sys ${ROOTFSDIR}/sys
- sudo mount --make-rslave ${ROOTFSDIR}/sys
-
- # Refresh /etc/resolv.conf
- sudo cp -L /etc/resolv.conf ${ROOTFSDIR}/etc
-
- # Install packages:
- E="${@ bb.utils.export_proxies(d)}"
- export DEBIAN_FRONTEND=noninteractive
- # To avoid Perl locale warnings:
- export LANG=C
- export LANGUAGE=C
- export LC_ALL=C
- sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update \
- -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \
- -o Dir::Etc::sourceparts="-" \
- -o APT::Get::List-Cleanup="0"
- # Add multiarch for cross-target
- if [ "${ROOTFS_ARCH}" != "${DISTRO_ARCH}" ]; then
- sudo -E chroot "$ROOTFSDIR" /usr/bin/dpkg --add-architecture ${DISTRO_ARCH}
- sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update
- fi
- sudo -E chroot "$ROOTFSDIR" \
- /usr/bin/apt-get ${APT_ARGS} --download-only $PACKAGES \
- ${IMAGE_TRANSIENT_PACKAGES}
- [ ${CLEAN} ] && sudo rm -f ${CLEAN_FILES}
- sudo -E chroot "$ROOTFSDIR" \
- /usr/bin/apt-get ${APT_ARGS} $PACKAGES
- for pkg in ${IMAGE_TRANSIENT_PACKAGES}; do
- sudo -E chroot "$ROOTFSDIR" \
- /usr/bin/apt-get ${APT_ARGS} $pkg
- done
- for pkg in ${@reverse_bb_array(d, "IMAGE_TRANSIENT_PACKAGES") or ""}; do
- sudo -E chroot "$ROOTFSDIR" \
- /usr/bin/apt-get purge --yes $pkg
- done
- if [ ${CLEAN} ]; then
- if [ ${KEEP_APT_CACHE} -eq 1 ]; then
- mkdir -p ${WORKDIR}/apt_cache
- sudo find ${ROOTFSDIR}/var/cache/apt/archives \
- -maxdepth 1 -name '*.deb' -execdir /bin/mv -t ${WORKDIR}/apt_cache '{}' '+'
- sudo chown -R $(whoami) ${WORKDIR}/apt_cache
- fi
- sudo -E chroot "$ROOTFSDIR" \
- /usr/bin/apt-get autoremove --purge --yes
- sudo -E chroot "$ROOTFSDIR" \
- /usr/bin/apt-get clean
- sudo "$ROOTFSDIR/chroot-setup.sh" "cleanup" "$ROOTFSDIR"
- sudo rm -rf "$ROOTFSDIR/chroot-setup.sh" "$ROOTFSDIR/var/lib/apt/lists/"*
- fi
-}
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 45b0350..0fc8ba8 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -1,14 +1,178 @@
# This software is a part of ISAR.
# Copyright (c) Siemens AG, 2019
+ROOTFS_ARCH ?= "${DISTRO_ARCH}"
+ROOTFS_DISTRO ?= "${DISTRO}"
+ROOTFS_PACKAGES ?= ""
+
# Features of the rootfs creation:
# available features are:
+# 'deploy-package-cache' - copy the package cache ${WORKDIR}/apt_cache
+# 'clean-package-cache' - delete package cache from rootfs
# 'finalize-rootfs' - delete files needed to chroot into the rootfs
ROOTFS_FEATURES ?= ""
+ROOTFS_APT_ARGS="install --yes -o Debug::pkgProblemResolver=yes"
+
+ROOTFS_CLEAN_FILES="/etc/hostname /etc/resolv.conf"
+
+# Useful environment variables:
+export E = "${@ bb.utils.export_proxies(d)}"
+export DEBIAN_FRONTEND = "noninteractive"
+# To avoid Perl locale warnings:
+export LANG = "C"
+export LANGUAGE = "C"
+export LC_ALL = "C"
+
+rootfs_do_mounts[weight] = "3"
+rootfs_do_mounts() {
+ sudo -s <<'EOSUDO'
+ mountpoint -q '${ROOTFSDIR}/dev' || \
+ mount --rbind /dev '${ROOTFSDIR}/dev'
+ mount --make-rslave '${ROOTFSDIR}/dev'
+ mountpoint -q '${ROOTFSDIR}/proc' || \
+ mount -t proc none '${ROOTFSDIR}/proc'
+ mountpoint -q '${ROOTFSDIR}/sys' || \
+ mount --rbind /sys '${ROOTFSDIR}/sys'
+ mount --make-rslave '${ROOTFSDIR}/sys'
+
+ # Mount isar-apt if the directory does not exist or if it is empty
+ # This prevents overwriting something that was copied there
+ if [ ! -e '${ROOTFSDIR}/isar-apt' ] || \
+ [ "$(find '${ROOTFSDIR}/isar-apt' -maxdepth 1 -mindepth 1 | wc -l)" = "0" ]
+ then
+ mkdir -p '${ROOTFSDIR}/isar-apt'
+ mountpoint -q '${ROOTFSDIR}/isar-apt' || \
+ mount --bind '${REPO_ISAR_DIR}/${DISTRO}' '${ROOTFSDIR}/isar-apt'
+ fi
+
+ # Mount base-apt if 'ISAR_USE_CACHED_BASE_REPO' is set
+ if [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" = 'True' ]
+ then
+ mkdir -p '${ROOTFSDIR}/base-apt'
+ mountpoint -q '${ROOTFSDIR}/base-apt' || \
+ mount --bind '${REPO_BASE_DIR}' '${ROOTFSDIR}/base-apt'
+ fi
+
+EOSUDO
+}
+
+rootfs_do_qemu() {
+ if [ '${@repr(d.getVar('ROOTFS_ARCH') == d.getVar('HOST_ARCH'))}' = 'False' ]
+ then
+ test -e '${ROOTFSDIR}/usr/bin/qemu-${QEMU_ARCH}-static' || \
+ sudo cp '/usr/bin/qemu-${QEMU_ARCH}-static' '${ROOTFSDIR}/usr/bin/qemu-${QEMU_ARCH}-static'
+ fi
+}
+
+BOOTSTRAP_SRC = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTFS_ARCH}-${DISTRO_ARCH}"
+BOOTSTRAP_SRC_${ROOTFS_ARCH} = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTFS_ARCH}"
+
+rootfs_prepare[weight] = "25"
+rootfs_prepare(){
+ sudo cp -Trpfx '${BOOTSTRAP_SRC}/' '${ROOTFSDIR}'
+}
+
+ROOTFS_CONFIGURE_COMMAND += "rootfs_configure_isar_apt"
+rootfs_configure_isar_apt[weight] = "2"
+rootfs_configure_isar_apt() {
+ sudo -s <<'EOSUDO'
+
+ mkdir -p '${ROOTFSDIR}/etc/apt/sources.list.d'
+ echo 'deb [trusted=yes] file:///isar-apt ${DEBDISTRONAME} main' > \
+ '${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list'
+
+ mkdir -p '${ROOTFSDIR}/etc/apt/preferences.d'
+ cat << EOF > '${ROOTFSDIR}/etc/apt/preferences.d/isar'
+Package: *
+Pin: release n=${DEBDISTRONAME}
+Pin-Priority: 1000
+EOF
+EOSUDO
+}
+
+ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_update"
+rootfs_install_pkgs_update[weight] = "5"
+rootfs_install_pkgs_update() {
+ sudo -E chroot '${ROOTFSDIR}' /usr/bin/apt-get update \
+ -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \
+ -o Dir::Etc::sourceparts="-" \
+ -o APT::Get::List-Cleanup="0"
+}
+ROOTFS_INSTALL_COMMAND += "rootfs_install_resolvconf"
+rootfs_install_resolvconf[weight] = "1"
+rootfs_install_resolvconf() {
+ sudo cp -rL /etc/resolv.conf '${ROOTFSDIR}/etc'
+}
+
+ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_download"
+rootfs_install_pkgs_download[weight] = "600"
+rootfs_install_pkgs_download() {
+ sudo -E chroot '${ROOTFSDIR}' \
+ /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only ${ROOTFS_PACKAGES}
+}
+
+ROOTFS_INSTALL_COMMAND += "${@ 'rootfs_install_clean_files' if (d.getVar('ROOTFS_CLEAN_FILES') or '').strip() else ''}"
+rootfs_install_clean_files[weight] = "2"
+rootfs_install_clean_files() {
+ sudo -E chroot '${ROOTFSDIR}' \
+ /bin/rm -f ${ROOTFS_CLEAN_FILES}
+}
+
+ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_install"
+rootfs_install_pkgs_install[weight] = "8000"
+rootfs_install_pkgs_install() {
+ sudo -E chroot "${ROOTFSDIR}" \
+ /usr/bin/apt-get ${ROOTFS_APT_ARGS} ${ROOTFS_PACKAGES}
+}
+
+do_rootfs_install[root_cleandirs] = "${ROOTFSDIR}"
+do_rootfs_install[vardeps] = "${ROOTFS_CONFIGURE_COMMAND} ${ROOTFS_INSTALL_COMMAND}"
+do_rootfs_install[depends] = "isar-bootstrap-${@'target' if d.getVar('ROOTFS_ARCH') == d.getVar('DISTRO_ARCH') else 'host'}:do_build isar-apt:do_cache_config"
+do_rootfs_install[deptask] = "do_deploy_deb"
+python do_rootfs_install() {
+ configure_cmds = (d.getVar("ROOTFS_CONFIGURE_COMMAND", True) or "").split()
+ install_cmds = (d.getVar("ROOTFS_INSTALL_COMMAND", True) or "").split()
+
+ # Mount after configure commands, so that they have time to copy
+ # 'isar-apt' (sdkchroot):
+ cmds = ['rootfs_prepare'] + configure_cmds + ['rootfs_do_mounts'] + install_cmds
+
+ stage_weights = [int(d.getVarFlag(i, 'weight', True) or "20")
+ for i in cmds]
+
+ progress_reporter = bb.progress.MultiStageProgressReporter(d, stage_weights)
+
+ for cmd in cmds:
+ progress_reporter.next_stage()
+ bb.build.exec_func(cmd, d)
+ progress_reporter.finish()
+}
+addtask rootfs_install before do_rootfs_postprocess after do_unpack
+
+ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'copy-package-cache', 'rootfs_postprocess_copy_package_cache', '', d)}"
+rootfs_postprocess_copy_package_cache() {
+ mkdir -p '${WORKDIR}/apt_cache'
+ sudo find '${ROOTFSDIR}/var/cache/apt/archives' \
+ -maxdepth 1 -name '*.deb' -execdir /bin/mv -t '${WORKDIR}/apt_cache' '{}' '+'
+ me="$(id -u):$(id -g)"
+ sudo chown -R "$me" '${WORKDIR}/apt_cache'
+}
+
+ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-package-cache', 'rootfs_postprocess_clean_package_cache', '', d)}"
+rootfs_postprocess_clean_package_cache() {
+ sudo -E chroot '${ROOTFSDIR}' \
+ /usr/bin/apt-get clean
+ sudo rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"*
+}
+
ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'finalize-rootfs', 'rootfs_postprocess_finalize', '', d)}"
rootfs_postprocess_finalize() {
sudo -s <<'EOSUDO'
+ test -e "${ROOTFSDIR}/chroot-setup.sh" && \
+ "${ROOTFSDIR}/chroot-setup.sh" "cleanup" "${ROOTFSDIR}"
+ rm -f "${ROOTFSDIR}/chroot-setup.sh"
+
test ! -e "${ROOTFSDIR}/usr/share/doc/qemu-user-static" && \
find "${ROOTFSDIR}/usr/bin" \
-maxdepth 1 -name 'qemu-*-static' -type f -delete
@@ -42,6 +206,12 @@ EOSUDO
do_rootfs_postprocess[vardeps] = "${ROOTFS_POSTPROCESS_COMMAND}"
python do_rootfs_postprocess() {
+ # Take care that its correctly mounted:
+ bb.build.exec_func('rootfs_do_mounts', d)
+ # Take care that qemu-*-static is available, since it could have been
+ # removed on a previous execution of this task:
+ bb.build.exec_func('rootfs_do_qemu', d)
+
cmds = d.getVar("ROOTFS_POSTPROCESS_COMMAND")
if cmds is None or not cmds.strip():
return
@@ -49,7 +219,7 @@ python do_rootfs_postprocess() {
for cmd in cmds:
bb.build.exec_func(cmd, d)
}
-addtask rootfs_postprocess before do_rootfs after do_rootfs_install
+addtask rootfs_postprocess before do_rootfs
python do_rootfs() {
"""Virtual task"""
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 0ec2e2b..ff4fd2e 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -285,6 +285,11 @@ isar_bootstrap() {
mount --make-rslave ${ROOTFSDIR}/sys
export DEBIAN_FRONTEND=noninteractive
+
+ if [ ${IS_HOST} ]; then
+ chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${DISTRO_ARCH}
+ fi
+
chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y
chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \
-o Debug::pkgProblemResolver=yes
diff --git a/meta/recipes-devtools/buildchroot/buildchroot-host.bb b/meta/recipes-devtools/buildchroot/buildchroot-host.bb
index e8f7f69..7daaa81 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot-host.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot-host.bb
@@ -6,6 +6,8 @@
DESCRIPTION = "Isar development filesystem for host"
require buildchroot.inc
+ROOTFS_ARCH = "${HOST_ARCH}"
+
PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
BUILDCHROOT_PREINSTALL ?= "make \
@@ -18,15 +20,5 @@ BUILDCHROOT_PREINSTALL ?= "make \
automake \
devscripts \
equivs \
- libc6:${DISTRO_ARCH}"
-
-# According to the wiki page:
-# https://wiki.debian.org/CrossToolchains
-BUILDCHROOT_PREINSTALL_append_armhf += "binutils-arm-linux-gnueabihf \
- crossbuild-essential-armhf"
-BUILDCHROOT_PREINSTALL_append_arm64 += "binutils-aarch64-linux-gnu \
- crossbuild-essential-arm64"
-
-
-PARAMS = "--host-arch --host-distro"
-do_build[depends] = "isar-apt:do_cache_config isar-bootstrap-host:do_bootstrap"
+ libc6:${DISTRO_ARCH} \
+ crossbuild-essential-${DISTRO_ARCH}"
diff --git a/meta/recipes-devtools/buildchroot/buildchroot-target.bb b/meta/recipes-devtools/buildchroot/buildchroot-target.bb
index ce6bd93..de58fc6 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot-target.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot-target.bb
@@ -20,5 +20,3 @@ BUILDCHROOT_PREINSTALL ?= "gcc \
devscripts \
equivs \
adduser"
-
-do_build[depends] = "isar-apt:do_cache_config isar-bootstrap-target:do_bootstrap"
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc b/meta/recipes-devtools/buildchroot/buildchroot.inc
index cdb4c07..bca6650 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.inc
+++ b/meta/recipes-devtools/buildchroot/buildchroot.inc
@@ -14,18 +14,25 @@ SRC_URI = "file://configscript.sh \
PV = "1.0"
PF = "${PN}-${DISTRO}-${DISTRO_ARCH}"
-inherit isar-bootstrap-helper
+inherit rootfs
BUILDCHROOT_DIR = "${WORKDIR}/rootfs"
+ROOTFSDIR = "${BUILDCHROOT_DIR}"
+ROOTFS_PACKAGES = "${BUILDCHROOT_PREINSTALL}"
+ROOTFS_CLEAN_FILES = ""
+
+rootfs_do_mounts_append() {
+ sudo -s <<'EOSUDO'
+ mkdir -p '${BUILDCHROOT_DIR}/downloads'
+ mountpoint -q '${BUILDCHROOT_DIR}/downloads' || \
+ mount --bind '${DL_DIR}' '${BUILDCHROOT_DIR}/downloads'
+EOSUDO
+}
-do_build[root_cleandirs] = "${BUILDCHROOT_DIR} \
- ${BUILDCHROOT_DIR}/isar-apt \
- ${BUILDCHROOT_DIR}/downloads \
- ${BUILDCHROOT_DIR}/home/builder"
-
-do_build() {
- setup_root_file_system ${PARAMS} ${BUILDCHROOT_DIR} ${BUILDCHROOT_PREINSTALL}
-
+ROOTFS_POSTPROCESS_COMMAND =+ "buildchroot_install_files"
+buildchroot_install_files() {
+ sudo mkdir -p "${BUILDCHROOT_DIR}/home/builder"
+ sudo mkdir -p "${BUILDCHROOT_DIR}/isar-apt"
# Install package builder script
sudo chmod -R a+rw "${BUILDCHROOT_DIR}/home/builder"
sudo install -m 755 -d ${BUILDCHROOT_DIR}/isar
@@ -39,5 +46,4 @@ do_build() {
GROUP_ID=$(id -g)
sudo chroot ${BUILDCHROOT_DIR} /configscript.sh $USER_ID $GROUP_ID
- sudo mount --bind ${DL_DIR} ${BUILDCHROOT_DIR}/downloads
}
diff --git a/meta/recipes-devtools/sdkchroot/sdkchroot.bb b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
index 249fa94..eb1676c 100644
--- a/meta/recipes-devtools/sdkchroot/sdkchroot.bb
+++ b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
@@ -12,10 +12,24 @@ SRC_URI = " \
file://configscript.sh \
file://README.sdk"
PV = "0.1"
-
-inherit isar-bootstrap-helper
PF = "${PN}-${ROOTFS_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
+TOOLCHAIN = "crossbuild-essential-${DISTRO_ARCH}"
+TOOLCHAIN_${HOST_ARCH} = "build-essential"
+TOOLCHAIN_i386 = "build-essential"
+
+inherit rootfs
+ROOTFS_ARCH = "${HOST_ARCH}"
+ROOTFSDIR = "${S}"
+ROOTFS_PACKAGES = "${SDKCHROOT_PREINSTALL} ${TOOLCHAIN}"
+ROOTFS_FEATURES += "copy-package-cache"
+
+python() {
+ if d.getVar("HOST_ARCH") not in ['i386', 'amd64']:
+ raise bb.parse.SkipRecipe("SDK doesn't support {} as host".format(
+ d.getVar("ROOTFS_ARCH")))
+}
+
SDKCHROOT_PREINSTALL := "debhelper \
autotools-dev \
dpkg \
@@ -28,29 +42,14 @@ SDKCHROOT_PREINSTALL := "debhelper \
S = "${WORKDIR}/rootfs"
-do_build[dirs] = "${DEPLOY_DIR_IMAGE}"
-do_build[root_cleandirs] = "${S} \
- ${S}/isar-apt"
-
-do_build[depends] = "isar-apt:do_cache_config isar-bootstrap-host:do_bootstrap"
-
-do_build() {
-
- if [ ${HOST_DISTRO} != "debian-stretch" ]; then
- bbfatal "SDK doesn't support ${HOST_DISTRO}"
- fi
- if [ ${HOST_ARCH} != "i386" -a ${HOST_ARCH} != "amd64" ]; then
- bbfatal "SDK doesn't support ${HOST_ARCH} as host"
- fi
-
- if [ ${HOST_ARCH} = ${DISTRO_ARCH} -o ${DISTRO_ARCH} = "i386" ]; then
- packages="${SDKCHROOT_PREINSTALL} build-essential"
- else
- packages="${SDKCHROOT_PREINSTALL} crossbuild-essential-${DISTRO_ARCH}"
- fi
-
- setup_root_file_system --copyisarapt --host-arch --host-distro "${S}" $packages
+ROOTFS_CONFIGURE_COMMAND += "rootfs_configure_isar_apt_dir"
+rootfs_configure_isar_apt_dir() {
+ # Copy isar-apt instead of mounting:
+ sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} ${ROOTFSDIR}/isar-apt
+}
+ROOTFS_POSTPROCESS_COMMAND =+ "sdkchroot_install_files"
+sdkchroot_install_files() {
# Configure root filesystem
sudo install -m 644 ${WORKDIR}/README.sdk ${S}
sudo install -m 755 ${WORKDIR}/configscript.sh ${S}
--
2.20.1
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH 2/6] meta: remove transient package support
2019-04-18 11:25 [PATCH 0/6] pre-processing pipeline and transient package replacement claudius.heine.ext
2019-04-18 11:25 ` [PATCH 1/6] split up isar-bootstrap helper and implement pre-process pipeline claudius.heine.ext
@ 2019-04-18 11:25 ` claudius.heine.ext
2019-04-18 11:25 ` [PATCH 3/6] meta/classes: add image-locales-extension class claudius.heine.ext
` (4 subsequent siblings)
6 siblings, 0 replies; 14+ messages in thread
From: claudius.heine.ext @ 2019-04-18 11:25 UTC (permalink / raw)
To: isar-users; +Cc: Claudius Heine
From: Claudius Heine <ch@denx.de>
Transient packages where meant for configuring images via packages,
which proved to not be fit for purpose.
Packages are meant to be shared across images, configuring it this way
would mean that every possible configuration permutation would need to
result in a separate package just so that no image accidentally installs
a configuration package meant for a different build/image.
Instead images need to be configured while its specific root file system
is created using the new root file system creation pipeline.
Signed-off-by: Claudius Heine <ch@denx.de>
---
meta/classes/image.bbclass | 4 +-
.../isar-cfg-localepurge/files/postinst | 15 ----
.../isar-cfg-localepurge.bb | 70 -------------------
.../isar-cfg-rootpw/files/postinst.tmpl | 19 -----
.../isar-cfg-rootpw/isar-cfg-rootpw.bb | 19 -----
5 files changed, 1 insertion(+), 126 deletions(-)
delete mode 100644 meta/recipes-support/isar-cfg-localepurge/files/postinst
delete mode 100644 meta/recipes-support/isar-cfg-localepurge/isar-cfg-localepurge.bb
delete mode 100644 meta/recipes-support/isar-cfg-rootpw/files/postinst.tmpl
delete mode 100644 meta/recipes-support/isar-cfg-rootpw/isar-cfg-rootpw.bb
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index c795414..89dc5e4 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -36,9 +36,7 @@ def cfg_script(d):
FILESPATH =. "${LAYERDIR_core}/conf/distro:"
SRC_URI += "${@ cfg_script(d) }"
-DEPENDS += "${IMAGE_INSTALL} ${IMAGE_TRANSIENT_PACKAGES}"
-
-IMAGE_TRANSIENT_PACKAGES += "isar-cfg-localepurge isar-cfg-rootpw"
+DEPENDS += "${IMAGE_INSTALL}"
ISAR_RELEASE_CMD_DEFAULT = "git -C ${LAYERDIR_core} describe --tags --dirty --match 'v[0-9].[0-9]*'"
ISAR_RELEASE_CMD ?= "${ISAR_RELEASE_CMD_DEFAULT}"
diff --git a/meta/recipes-support/isar-cfg-localepurge/files/postinst b/meta/recipes-support/isar-cfg-localepurge/files/postinst
deleted file mode 100644
index 9dfd6f9..0000000
--- a/meta/recipes-support/isar-cfg-localepurge/files/postinst
+++ /dev/null
@@ -1,15 +0,0 @@
-#!/bin/sh
-set -e
-
-cat /usr/lib/isar-cfg-localepurge/locale.gen \
- >> /etc/locale.gen
-cat /usr/lib/isar-cfg-localepurge/locale.default \
- > /etc/default/locale
-cat /usr/lib/isar-cfg-localepurge/locale.nopurge \
- > /etc/locale.nopurge
-
-debconf-set-selections /usr/lib/isar-cfg-localepurge/locale.debconf
-
-dpkg-reconfigure -f noninteractive locales
-
-localepurge
diff --git a/meta/recipes-support/isar-cfg-localepurge/isar-cfg-localepurge.bb b/meta/recipes-support/isar-cfg-localepurge/isar-cfg-localepurge.bb
deleted file mode 100644
index 62b4b2d..0000000
--- a/meta/recipes-support/isar-cfg-localepurge/isar-cfg-localepurge.bb
+++ /dev/null
@@ -1,70 +0,0 @@
-# This software is a part of ISAR.
-
-DESCRIPTION = "Isar configuration package for locale and localepurge"
-MAINTAINER = "isar-users <isar-users@googlegroups.com>"
-DEBIAN_DEPENDS = "localepurge"
-
-SRC_URI = "file://postinst"
-
-inherit dpkg-raw
-
-LOCALE_GEN ?= "en_US.UTF-8 UTF-8\n\
- en_US ISO-8859-1\n"
-LOCALE_DEFAULT ?= "en_US.UTF-8"
-
-def get_locale_gen(d):
- locale_gen = d.getVar("LOCALE_GEN", True) or ""
- return '\n'.join(sorted(set(i.strip()
- for i in locale_gen.split('\\n')
- if i.strip())))
-
-def get_dc_locale_gen(d):
- locale_gen = d.getVar("LOCALE_GEN", True) or ""
- return ', '.join(sorted(set(i.strip()
- for i in locale_gen.split('\\n')
- if i.strip())))
-
-def get_nopurge(d):
- locale_gen = d.getVar("LOCALE_GEN", True) or ""
- return '\n'.join(sorted(set(i.strip()
- for j in locale_gen.split('\\n')
- if j.strip()
- for i in (j.split()[0].split("_")[0],
- j.split()[0].split(".")[0],
- j.split()[0]))))
-
-do_gen_config[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
-do_gen_config() {
- cat<<-__EOF__ > ${WORKDIR}/locale.gen
- ${@get_locale_gen(d)}
- __EOF__
- cat<<-__EOF__ > ${WORKDIR}/locale.debconf
- locales locales/locales_to_be_generated multiselect ${@get_dc_locale_gen(d)}
- locales locales/default_environment_locale select ${LOCALE_DEFAULT}
- __EOF__
- cat<<-__EOF__ > ${WORKDIR}/locale.default
- LANG=${LOCALE_DEFAULT}
- __EOF__
- cat<<-__EOF__ > ${WORKDIR}/locale.nopurge
- #USE_DPKG
- MANDELETE
- DONTBOTHERNEWLOCALE
- #SHOWFREEDSPACE
- #QUICKNDIRTYCALC
- #VERBOSE
- ${@get_nopurge(d)}
- __EOF__
-}
-addtask gen_config after do_unpack before do_install
-
-do_install() {
- install -v -d ${D}/usr/lib/${PN}
- install -v -m 644 ${WORKDIR}/locale.debconf \
- ${D}/usr/lib/${PN}/locale.debconf
- install -v -m 644 ${WORKDIR}/locale.gen \
- ${D}/usr/lib/${PN}/locale.gen
- install -v -m 644 ${WORKDIR}/locale.default \
- ${D}/usr/lib/${PN}/locale.default
- install -v -m 644 ${WORKDIR}/locale.nopurge \
- ${D}/usr/lib/${PN}/locale.nopurge
-}
diff --git a/meta/recipes-support/isar-cfg-rootpw/files/postinst.tmpl b/meta/recipes-support/isar-cfg-rootpw/files/postinst.tmpl
deleted file mode 100644
index ca08a41..0000000
--- a/meta/recipes-support/isar-cfg-rootpw/files/postinst.tmpl
+++ /dev/null
@@ -1,19 +0,0 @@
-#!/bin/sh
-set -e
-
-if ! grep -q 'root:\*:' /etc/shadow; then
- echo "ERROR:isar-cfg-rootpw: root password was set by a different package" >&2
- exit -1
-fi
-
-if [ -n '${CFG_ROOT_PW}' ]; then
- echo 'root:${CFG_ROOT_PW}' | chpasswd -e
-else
- passwd -d root
-fi
-
-if [ '${CFG_ROOT_LOCKED}' = "1" ]; then
- # Lock the account after setting the password, since unlocking it at some
- # point later would set it to the back to the previous one.
- passwd -l root
-fi
diff --git a/meta/recipes-support/isar-cfg-rootpw/isar-cfg-rootpw.bb b/meta/recipes-support/isar-cfg-rootpw/isar-cfg-rootpw.bb
deleted file mode 100644
index adee3b5..0000000
--- a/meta/recipes-support/isar-cfg-rootpw/isar-cfg-rootpw.bb
+++ /dev/null
@@ -1,19 +0,0 @@
-# This software is a part of ISAR.
-
-DESCRIPTION = "Isar configuration package for root password"
-MAINTAINER = "isar-users <isar-users@googlegroups.com>"
-DEBIAN_DEPENDS = "passwd"
-
-SRC_URI = "file://postinst.tmpl"
-
-TEMPLATE_FILES = "postinst.tmpl"
-TEMPLATE_VARS = "CFG_ROOT_PW CFG_ROOT_LOCKED"
-
-CFG_ROOT_PW ??= ""
-CFG_ROOT_LOCKED ??= "0"
-
-inherit dpkg-raw
-
-do_install() {
- echo "intentionally left blank"
-}
--
2.20.1
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH 3/6] meta/classes: add image-locales-extension class
2019-04-18 11:25 [PATCH 0/6] pre-processing pipeline and transient package replacement claudius.heine.ext
2019-04-18 11:25 ` [PATCH 1/6] split up isar-bootstrap helper and implement pre-process pipeline claudius.heine.ext
2019-04-18 11:25 ` [PATCH 2/6] meta: remove transient package support claudius.heine.ext
@ 2019-04-18 11:25 ` claudius.heine.ext
2019-04-18 11:25 ` [PATCH 4/6] meta/classes: add image-account-extension class claudius.heine.ext
` (3 subsequent siblings)
6 siblings, 0 replies; 14+ messages in thread
From: claudius.heine.ext @ 2019-04-18 11:25 UTC (permalink / raw)
To: isar-users; +Cc: Claudius Heine
From: Claudius Heine <ch@denx.de>
This class allows to configure the locales of the image and also uses
'localepurge' to remove unwanted locales.
The locale can be configured the same way as with the
isar-cfg-localepurge package, via the `LOCALE_GEN` and `LOCALE_DEFAULT`
variables.
Signed-off-by: Claudius Heine <ch@denx.de>
---
meta/classes/image-locales-extension.bbclass | 97 ++++++++++++++++++++
meta/classes/image.bbclass | 1 +
meta/classes/rootfs.bbclass | 4 +
3 files changed, 102 insertions(+)
create mode 100644 meta/classes/image-locales-extension.bbclass
diff --git a/meta/classes/image-locales-extension.bbclass b/meta/classes/image-locales-extension.bbclass
new file mode 100644
index 0000000..3c0758f
--- /dev/null
+++ b/meta/classes/image-locales-extension.bbclass
@@ -0,0 +1,97 @@
+# This software is a part of ISAR.
+# Copyright (C) Siemens AG, 2019
+#
+# SPDX-License-Identifier: MIT
+#
+# This class extends the image.bbclass for setting locales and purging unneeded
+# ones.
+
+LOCALE_GEN ?= "en_US.UTF-8 UTF-8\n\
+ en_US ISO-8859-1\n"
+LOCALE_DEFAULT ?= "en_US.UTF-8"
+
+def get_locale_gen(d, sep='\n'):
+ locale_gen = d.getVar("LOCALE_GEN", True) or ""
+ return sep.join(sorted(set(i.strip()
+ for i in locale_gen.split('\\n')
+ if i.strip())))
+
+def get_nopurge(d):
+ locale_gen = d.getVar("LOCALE_GEN", True) or ""
+ return '\n'.join(sorted(set(i.strip()
+ for j in locale_gen.split('\\n')
+ if j.strip()
+ for i in (j.split()[0].split("_")[0],
+ j.split()[0].split(".")[0],
+ j.split()[0]))))
+
+ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN += "image_install_localepurge_download"
+image_install_localepurge_download[weight] = "40"
+image_install_localepurge_download() {
+ sudo -E chroot '${ROOTFSDIR}' \
+ /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only localepurge
+}
+
+ROOTFS_INSTALL_COMMAND += "image_install_localepurge_install"
+image_install_localepurge_install[weight] = "700"
+image_install_localepurge_install() {
+
+ # Generate locale and localepurge configuration:
+ cat<<__EOF__ > ${WORKDIR}/locale.gen
+${@get_locale_gen(d)}
+__EOF__
+ cat<<__EOF__ > ${WORKDIR}/locale.debconf
+locales locales/locales_to_be_generated multiselect ${@get_locale_gen(d, ', ')}
+locales locales/default_environment_locale select ${LOCALE_DEFAULT}
+__EOF__
+ cat<<__EOF__ > ${WORKDIR}/locale.default
+LANG=${LOCALE_DEFAULT}
+__EOF__
+ cat<<__EOF__ > ${WORKDIR}/locale.nopurge
+#USE_DPKG
+MANDELETE
+DONTBOTHERNEWLOCALE
+#SHOWFREEDSPACE
+#QUICKNDIRTYCALC
+#VERBOSE
+${@get_nopurge(d)}
+__EOF__
+
+ # Install configuration into image:
+ sudo -E -s <<'EOSUDO'
+ cat '${WORKDIR}/locale.gen' >> '${ROOTFSDIR}/etc/locale.gen'
+ cat '${WORKDIR}/locale.default' > '${ROOTFSDIR}/etc/default/locale'
+ cat '${WORKDIR}/locale.nopurge' > '${ROOTFSDIR}/etc/locale.nopurge'
+ cat '${WORKDIR}/locale.debconf' > '${ROOTFSDIR}/tmp/locale.debconf'
+
+ # Enter image and trigger locales config and localepurge:
+ chroot '${ROOTFSDIR}' /bin/sh <<'EOSH'
+ localepurge_state='i'
+ if dpkg -s localepurge 2>/dev/null >&2
+ then
+ echo 'localepurge was installed (leaving it installed later)'
+ else
+ localepurge_state='p'
+ echo 'localepurge was not installed (removing it later)'
+ apt-get ${ROOTFS_APT_ARGS} localepurge
+ fi
+
+ echo 'running locale debconf-set-selections'
+ debconf-set-selections /tmp/locale.debconf
+ rm -f '/tmp/locale.debconf'
+
+ echo 'reconfigure locales'
+ dpkg-reconfigure -f noninteractive locales
+
+ echo 'running localepurge'
+ localepurge
+
+ if [ "$localepurge_state" = 'p' ]
+ then
+ echo removing localepurge...
+ apt-get purge --yes localepurge
+ apt-get autoremove --purge --yes
+ fi
+EOSH
+EOSUDO
+}
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 89dc5e4..c1ae4f6 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -60,6 +60,7 @@ inherit image-sdk-extension
inherit image-cache-extension
inherit image-tools-extension
inherit image-postproc-extension
+inherit image-locales-extension
# Extra space for rootfs in MB
ROOTFS_EXTRA ?= "64"
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 0fc8ba8..5736eee 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -99,6 +99,7 @@ rootfs_install_pkgs_update() {
-o Dir::Etc::sourceparts="-" \
-o APT::Get::List-Cleanup="0"
}
+
ROOTFS_INSTALL_COMMAND += "rootfs_install_resolvconf"
rootfs_install_resolvconf[weight] = "1"
rootfs_install_resolvconf() {
@@ -112,6 +113,9 @@ rootfs_install_pkgs_download() {
/usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only ${ROOTFS_PACKAGES}
}
+ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN ??= ""
+ROOTFS_INSTALL_COMMAND += "${ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN}"
+
ROOTFS_INSTALL_COMMAND += "${@ 'rootfs_install_clean_files' if (d.getVar('ROOTFS_CLEAN_FILES') or '').strip() else ''}"
rootfs_install_clean_files[weight] = "2"
rootfs_install_clean_files() {
--
2.20.1
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH 4/6] meta/classes: add image-account-extension class
2019-04-18 11:25 [PATCH 0/6] pre-processing pipeline and transient package replacement claudius.heine.ext
` (2 preceding siblings ...)
2019-04-18 11:25 ` [PATCH 3/6] meta/classes: add image-locales-extension class claudius.heine.ext
@ 2019-04-18 11:25 ` claudius.heine.ext
2019-04-18 11:25 ` [PATCH 5/6] doc: update description of image customization claudius.heine.ext
` (2 subsequent siblings)
6 siblings, 0 replies; 14+ messages in thread
From: claudius.heine.ext @ 2019-04-18 11:25 UTC (permalink / raw)
To: isar-users; +Cc: Claudius Heine
From: Claudius Heine <ch@denx.de>
This class allows to configure user and group accounts of the image.
Groups or users that should be configured/created are added into the
`GROUPS` or `USERS` variable.
The configuration itself is then added to each groups or users
`GROUP_<groupname>` or `USER_<username>` flags.
The flags available for groups are `gid` and `flags`. The `flags`
variable contains some additional options for the group. With this patch
only `system` is supported for groups, allowing to create groups with
`groupadd` with the `--system` parameter.
The flags available for users are `password`, `expire`, `inactive`,
`uid`, `gid`, `comment`, `home`, `shell`, `groups` and `flags`. The
additional flags for users are `nohome`, `system` and
`allowemptypassword`.
Signed-off-by: Claudius Heine <ch@denx.de>
---
meta/classes/image-account-extension.bbclass | 253 +++++++++++++++++++
meta/classes/image.bbclass | 1 +
2 files changed, 254 insertions(+)
create mode 100644 meta/classes/image-account-extension.bbclass
diff --git a/meta/classes/image-account-extension.bbclass b/meta/classes/image-account-extension.bbclass
new file mode 100644
index 0000000..f1eec50
--- /dev/null
+++ b/meta/classes/image-account-extension.bbclass
@@ -0,0 +1,253 @@
+# This software is a part of ISAR.
+# Copyright (C) Siemens AG, 2019
+#
+# SPDX-License-Identifier: MIT
+#
+# This class extends the image.bbclass for creating user accounts and groups.
+
+USERS ??= ""
+
+#USERS += "root"
+#USER_root[password] = "" # Encrypted password
+#USER_root[expire] = ""
+#USER_root[inactive] = ""
+#USER_root[uid] = ""
+#USER_root[gid] = "" # If first character is a number: gid, otherwise groupname
+#USER_root[comment] = "The ultimate root user"
+#USER_root[home] = "/home/root"
+#USER_root[shell] = "/bin/sh"
+#USER_root[groups] = "audio video"
+#USER_root[flags] = "nohome system allowemptypassword"
+
+GROUPS ??= ""
+
+#GROUPS += "root"
+#GROUP_root[gid] = ""
+#GROUP_root[flags] = "system"
+
+def gen_accounts_array(d, listname, entryname, flags, verb_flags=None):
+ from itertools import chain
+
+ entries = (d.getVar(listname, True) or "").split()
+ return " ".join(
+ ":".join(
+ chain(
+ (entry,),
+ (
+ (",".join(
+ (
+ d.getVarFlag(entryname + "_" + entry, flag, True) or ""
+ ).split()
+ ) if flag not in (verb_flags or []) else (
+ d.getVarFlag(entryname + "_" + entry, flag, True) or ""
+ )).replace(":","=")
+ for flag in flags
+ ),
+ )
+ )
+ for entry in entries
+ )
+
+# List of space separated entries, where each entry has the format:
+# username:encryptedpassword:expiredate:inactivenumber:userid:groupid:comment:homedir:shell:group1,group2:flag1,flag2
+IMAGE_ACCOUNTS_USERS =+ "${@gen_accounts_array(d, 'USERS', 'USER', ['password', 'expire', 'inactive', 'uid', 'gid', 'comment', 'home', 'shell', 'groups', 'flags'], ['password', 'comment', 'home', 'shell'])}"
+
+# List of space separated entries, where each entry has the format:
+# groupname:groupid:flag1,flag2
+IMAGE_ACCOUNTS_GROUPS =+ "${@gen_accounts_array(d, 'GROUPS', 'GROUP', ['gid', 'flags'])}"
+
+ROOTFS_CONFIGURE_COMMAND += "image_configure_accounts"
+image_configure_accounts[weight] = "3"
+image_configure_accounts() {
+ # Create groups
+ # Add space to the end of the list:
+ list='${@" ".join(d.getVar('IMAGE_ACCOUNTS_GROUPS', True).split())} '
+ while true; do
+ # Pop first group entry:
+ list_rest="${list#*:*:* }"
+ entry="${list%%${list_rest}}"
+ list="${list_rest}"
+
+ if [ -z "${entry}" ]; then
+ break
+ fi
+
+ # Add colon to the end of the entry and remove trailing space:
+ entry="${entry% }:"
+
+ # Decode entries:
+ name="${entry%%:*}"
+ entry="${entry#${name}:}"
+
+ gid="${entry%%:*}"
+ entry="${entry#${gid}:}"
+
+ flags="${entry%%:*}"
+ entry="${entry#${flags}:}"
+
+ flags=",${flags}," # Needed for searching for substrings
+
+ # Check if user already exists:
+ if grep -q "^${name}:" '${ROOTFSDIR}/etc/group'; then
+ exists="y"
+ else
+ exists="n"
+ fi
+
+ # Create arguments:
+ set -- # clear arguments
+
+ if [ -n "$gid" ]; then
+ set -- "$@" --gid "$gid"
+ fi
+
+ if [ "n" = "$exists" ]; then
+ if [ "${flags}" != "${flags%*,system,*}" ]; then
+ set -- "$@" --system
+ fi
+ fi
+
+ # Create or modify groups:
+ if [ "y" = "$exists" ]; then
+ if [ -z "$@" ]; then
+ echo "Do not execute groupmod (no changes)."
+ else
+ echo "Execute groupmod with \"$@\" for \"$name\""
+ sudo -E chroot '${ROOTFSDIR}' \
+ /usr/sbin/groupmod "$@" "$name"
+ fi
+ else
+ echo "Execute groupadd with \"$@\" for \"$name\""
+ sudo -E chroot '${ROOTFSDIR}' \
+ /usr/sbin/groupadd "$@" "$name"
+ fi
+ done
+
+ # Create users
+ list='${@" ".join(d.getVar('IMAGE_ACCOUNTS_USERS', True).split())} '
+ while true; do
+ # Pop first user entry:
+ list_rest="${list#*:*:*:*:*:*:*:*:*:*:* }"
+ entry="${list%%${list_rest}}"
+ list="${list_rest}"
+
+ if [ -z "${entry}" ]; then
+ break
+ fi
+
+ # Add colon to the end of the entry and remove trailing space:
+ entry="${entry% }:"
+
+ # Decode entries:
+ name="${entry%%:*}"
+ entry="${entry#${name}:}"
+
+ password="${entry%%:*}"
+ entry="${entry#${password}:}"
+
+ expire="${entry%%:*}"
+ entry="${entry#${expire}:}"
+
+ inactive="${entry%%:*}"
+ entry="${entry#${inactive}:}"
+
+ uid="${entry%%:*}"
+ entry="${entry#${uid}:}"
+
+ gid="${entry%%:*}"
+ entry="${entry#${gid}:}"
+
+ comment="${entry%%:*}"
+ entry="${entry#${comment}:}"
+
+ home="${entry%%:*}"
+ entry="${entry#${home}:}"
+
+ shell="${entry%%:*}"
+ entry="${entry#${shell}:}"
+
+ groups="${entry%%:*}"
+ entry="${entry#${groups}:}"
+
+ flags="${entry%%:*}"
+ entry="${entry#${flags}:}"
+
+ flags=",${flags}," # Needed for searching for substrings
+
+ # Check if user already exists:
+ if grep -q "^${name}:" '${ROOTFSDIR}/etc/passwd'; then
+ exists="y"
+ else
+ exists="n"
+ fi
+
+ # Create arguments:
+ set -- # clear arguments
+
+ if [ -n "$expire" ]; then
+ set -- "$@" --expiredate "$expire"
+ fi
+
+ if [ -n "$inactive" ]; then
+ set -- "$@" --inactive "$inactive"
+ fi
+
+ if [ -n "$uid" ]; then
+ set -- "$@" --uid "$uid"
+ fi
+
+ if [ -n "$gid" ]; then
+ set -- "$@" --gid "$gid"
+ fi
+
+ if [ -n "$comment" ]; then
+ set -- "$@" --comment "$comment"
+ fi
+
+ if [ -n "$home" ]; then
+ if [ "y" = "$exists" ]; then
+ set -- "$@" --home "$home" --move-home
+ else
+ set -- "$@" --home-dir "$home"
+ fi
+ fi
+
+ if [ -n "$shell" ]; then
+ set -- "$@" --shell "$shell"
+ fi
+
+ if [ -n "$groups" ]; then
+ set -- "$@" --groups "$groups"
+ fi
+
+ if [ "n" = "$exists" ]; then
+ if [ "${flags}" != "${flags%*,system,*}" ]; then
+ set -- "$@" --system
+ fi
+ if [ "${flags}" != "${flags%*,nohome,*}" ]; then
+ set -- "$@" --no-create-home
+ fi
+ fi
+
+ # Create or modify users:
+ if [ "y" = "$exists" ]; then
+ if [ -z "$@" ]; then
+ echo "Do not execute usermod (no changes)."
+ else
+ echo "Execute usermod with \"$@\" for \"$name\""
+ sudo -E chroot '${ROOTFSDIR}' \
+ /usr/sbin/usermod "$@" "$name"
+ fi
+ else
+ echo "Execute useradd with \"$@\" for \"$name\""
+ sudo -E chroot '${ROOTFSDIR}' \
+ /usr/sbin/useradd "$@" "$name"
+ fi
+
+ # Set password:
+ if [ -n "$password" -o "${flags}" != "${flags%*,allowemptypassword,*}" ]; then
+ printf '%s:%s' "$name" "$password" | sudo chroot '${ROOTFSDIR}' \
+ /usr/sbin/chpasswd -e
+ fi
+ done
+}
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index c1ae4f6..67c86b7 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -61,6 +61,7 @@ inherit image-cache-extension
inherit image-tools-extension
inherit image-postproc-extension
inherit image-locales-extension
+inherit image-account-extension
# Extra space for rootfs in MB
ROOTFS_EXTRA ?= "64"
--
2.20.1
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH 5/6] doc: update description of image customization
2019-04-18 11:25 [PATCH 0/6] pre-processing pipeline and transient package replacement claudius.heine.ext
` (3 preceding siblings ...)
2019-04-18 11:25 ` [PATCH 4/6] meta/classes: add image-account-extension class claudius.heine.ext
@ 2019-04-18 11:25 ` claudius.heine.ext
2019-04-18 11:25 ` [PATCH 6/6] doc: some fixes claudius.heine.ext
2019-04-22 14:09 ` [PATCH 0/6] pre-processing pipeline and transient package replacement Maxim Yu. Osipov
6 siblings, 0 replies; 14+ messages in thread
From: claudius.heine.ext @ 2019-04-18 11:25 UTC (permalink / raw)
To: isar-users; +Cc: Claudius Heine
From: Claudius Heine <ch@denx.de>
Signed-off-by: Claudius Heine <ch@denx.de>
---
doc/user_manual.md | 50 ++++++++++++++++++++++++++++++++++++----------
1 file changed, 39 insertions(+), 11 deletions(-)
diff --git a/doc/user_manual.md b/doc/user_manual.md
index ba57319..b9a40f9 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -314,15 +314,11 @@ Some other variables include:
- `IMAGE_INSTALL` - The list of custom packages to build and install to target image, please refer to relative chapter for more information.
- `BB_NUMBER_THREADS` - The number of `bitbake` jobs that can be run in parallel. Please set this option according your host CPU cores number.
- - `LOCALE_GEN` - A `\n` seperated list of `/etc/locale.gen` entries desired on the target.
- - `LOCALE_DEFAULT` - The default locale used for the `LANG` and `LANGUAGE` variable in `/etc/locale`.
- `HOST_DISTRO` - The distro to use for SDK root filesystem (so far limited only to `debian-stretch`). This variable is optional.
- `HOST_ARCH` - The Debian architecture of SDK root filesystem (e.g., `amd64`). By default set to current Debian host architecture. This variable is optional.
- `HOST_DISTRO_APT_SOURCES` - List of apt source files for SDK root filesystem. This variable is optional.
- `HOST_DISTRO_APT_PREFERENCES` - List of apt preference files for SDK root filesystem. This variable is optional.
- `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the default URI in the format `ftp.debian.org my.preferred.mirror`. This variable is optional.
- - `CFG_ROOT_PW` - The encrypted root password to be set. To encrypt password use `mkpasswd`. You find `mkpasswd` in the `whois` package of Debian. If the variable is empty, root login is passwordless.
- - `CFG_ROOT_LOCKED` - If set to `1` the root account will be locked.
---
@@ -493,17 +489,49 @@ Isar contains additional image type classes that can be used as reference:
## Customize and configure image
-Customization and configuration of an image should be done via packages, see below.
+Customization and configuration of an image can be done in two ways:
+
+ 1. Creating and adding a configuration package to `IMAGE_INSTALL`, or
+ 2. Changing the bitbake variables of the image recipe.
+
+In cases where configuration is not image specific, does not contain any secrets and can be shared between images, creating and adding a configuration package to `IMAGE_INSTALL` is the right option. This should be the case with most product specific configuration files.
+
+In cases where the configuration would contain secrets like user passwords, that would be world readable in `postinst`, etc. script files, some image extensions where created, that allow customization of those options from within the image recipe using bitbake variables. (e.g. user and group management and locale settings)
+
+### Locale configuration
+
+Two variables can be used to configure the locale installed on a image:
+
+ - `LOCALE_GEN` - A `\n` seperated list of `/etc/locale.gen` entries desired on the target.
+ - `LOCALE_DEFAULT` - The default locale used for the `LANG` and `LANGUAGE` variable in `/etc/locale`.
+
+### User and group configuration
+
+Groups can be created or modified using the `GROUPS` and `GROUP_<groupname>` variable or their flags.
+
+The `GROUPS` variable contains a space separated list of group names that should be modified or created. Each entry of this variable should have a corresponding `GROUP_<groupname>` variable.
-Adding those configuration packages to the image can be done in two ways:
+The `GROUP_<groupname>` variable contains the settings of a group named `groupname` in its flags. The following flags can be used:
- 1. Simply adding the package to `IMAGE_INSTALL`, like any other isar created package, or
- 2. Adding the package to `IMAGE_TRANSIENT_PACKAGES`.
+ - `gid` - The numeric group id.
+ - `flags` - A list of additional flags of the group. Those are the currently recognized flags:
+ - `system` - The group is created using the `--system` parameter.
-In most cases adding the configuration package to `IMAGE_INSTALL` is the right option.
+The `USERS` and `USER_<username>` variable works similar to the `GROUPS` and `GROUP_<groupname>` variable. The difference are the accepted flags of the `USER_<username>` variable. It accepts the following flags:
-In cases were the configuration script of the package has some external dependencies, that should not be part of the final image, then `IMAGE_TRANSIENT_PACKAGES` is the right option.
-Packages in the `IMAGE_TRANSIENT_PACKAGES` variable are installed to the image and purged in the next step. If such a configuration package deploys file as part of their content, then those files will be removed as well.
+ - `password` - The crypt(3) encrypted password. To encrypt a password use for example `mkpasswd` or `openssl passwd -6`. You can find `mkpasswd` in the `whois` package of Debian.
+ - `expire` - A `YYYY-MM-DD` formatted date on which the user account will be disabled. (see useradd(8))
+ - `inactive` - The number of days after a password expires until the account is permanently disabled. (see useradd(8))
+ - `uid` - The numeric user id.
+ - `gid` - The numeric group id or group name of this users initial login group.
+ - `comment` - This users comment field. Commonly the following format `full name,room number,work phone number,home phone number,other entry`.
+ - `home` - This users home directory
+ - `shell` - This users login shell
+ - `groups` - A space separated list of groups this user is a member of.
+ - `flags` - A list of additional flags of the user:
+ - `nohome` - `useradd` will be called with `-M` to prevent creation of the users home directory.
+ - `system` - `useradd` will be called with `--system`.
+ - `allowemptypassword` - Even if the `password` flag is empty, it will still be set. This results in a login without password.
---
--
2.20.1
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH 6/6] doc: some fixes
2019-04-18 11:25 [PATCH 0/6] pre-processing pipeline and transient package replacement claudius.heine.ext
` (4 preceding siblings ...)
2019-04-18 11:25 ` [PATCH 5/6] doc: update description of image customization claudius.heine.ext
@ 2019-04-18 11:25 ` claudius.heine.ext
2019-04-22 14:09 ` [PATCH 0/6] pre-processing pipeline and transient package replacement Maxim Yu. Osipov
6 siblings, 0 replies; 14+ messages in thread
From: claudius.heine.ext @ 2019-04-18 11:25 UTC (permalink / raw)
To: isar-users; +Cc: Claudius Heine
From: Claudius Heine <ch@denx.de>
Signed-off-by: Claudius Heine <ch@denx.de>
---
doc/user_manual.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/user_manual.md b/doc/user_manual.md
index b9a40f9..af143df 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -314,7 +314,7 @@ Some other variables include:
- `IMAGE_INSTALL` - The list of custom packages to build and install to target image, please refer to relative chapter for more information.
- `BB_NUMBER_THREADS` - The number of `bitbake` jobs that can be run in parallel. Please set this option according your host CPU cores number.
- - `HOST_DISTRO` - The distro to use for SDK root filesystem (so far limited only to `debian-stretch`). This variable is optional.
+ - `HOST_DISTRO` - The distro to use for SDK root filesystem. This variable is optional.
- `HOST_ARCH` - The Debian architecture of SDK root filesystem (e.g., `amd64`). By default set to current Debian host architecture. This variable is optional.
- `HOST_DISTRO_APT_SOURCES` - List of apt source files for SDK root filesystem. This variable is optional.
- `HOST_DISTRO_APT_PREFERENCES` - List of apt preference files for SDK root filesystem. This variable is optional.
--
2.20.1
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 0/6] pre-processing pipeline and transient package replacement
2019-04-18 11:25 [PATCH 0/6] pre-processing pipeline and transient package replacement claudius.heine.ext
` (5 preceding siblings ...)
2019-04-18 11:25 ` [PATCH 6/6] doc: some fixes claudius.heine.ext
@ 2019-04-22 14:09 ` Maxim Yu. Osipov
2019-04-24 7:11 ` Claudius Heine
6 siblings, 1 reply; 14+ messages in thread
From: Maxim Yu. Osipov @ 2019-04-22 14:09 UTC (permalink / raw)
To: claudius.heine.ext, isar-users; +Cc: Claudius Heine
The patch set doesn't apply against current 'next' (it would be more
convenient to mention that this patchset depends on your previous series
[PATCH v2 0/8] Cleanup rootfs creation).
Maxim.
On 4/18/19 1:25 PM, claudius.heine.ext@siemens.com wrote:
> From: Claudius Heine <ch@denx.de>
>
> Hi,
>
> here is the tested, documented and slightly modified version of the previous
> RFC patch set.
>
> The CI has shown that the RFC patchset was not rebuildable. This
> was patched by consolidating the `do_rootfs_prepare`, `do_roofs_configure`
> and `do_rootfs_install` into just one task `do_rootfs_install` that now calls
> the functions of both previous tasks as well as the `rootfs_prepare` function
> that previously was its own task `do_rootfs_prepare`.
>
> Otherwise I added documentation and more verbose commit messages.
>
> regards,
> Claudius
>
> Claudius Heine (6):
> split up isar-bootstrap helper and implement pre-process pipeline
> meta: remove transient package support
> meta/classes: add image-locales-extension class
> meta/classes: add image-account-extension class
> doc: update description of image customization
> doc: some fixes
>
> doc/user_manual.md | 52 +++-
> meta/classes/image-account-extension.bbclass | 253 ++++++++++++++++++
> meta/classes/image-locales-extension.bbclass | 97 +++++++
> meta/classes/image.bbclass | 30 +--
> meta/classes/isar-bootstrap-helper.bbclass | 143 ----------
> meta/classes/rootfs.bbclass | 176 +++++++++++-
> .../isar-bootstrap/isar-bootstrap.inc | 5 +
> .../buildchroot/buildchroot-host.bb | 16 +-
> .../buildchroot/buildchroot-target.bb | 2 -
> .../buildchroot/buildchroot.inc | 26 +-
> meta/recipes-devtools/sdkchroot/sdkchroot.bb | 47 ++--
> .../isar-cfg-localepurge/files/postinst | 15 --
> .../isar-cfg-localepurge.bb | 70 -----
> .../isar-cfg-rootpw/files/postinst.tmpl | 19 --
> .../isar-cfg-rootpw/isar-cfg-rootpw.bb | 19 --
> 15 files changed, 624 insertions(+), 346 deletions(-)
> create mode 100644 meta/classes/image-account-extension.bbclass
> create mode 100644 meta/classes/image-locales-extension.bbclass
> delete mode 100644 meta/classes/isar-bootstrap-helper.bbclass
> delete mode 100644 meta/recipes-support/isar-cfg-localepurge/files/postinst
> delete mode 100644 meta/recipes-support/isar-cfg-localepurge/isar-cfg-localepurge.bb
> delete mode 100644 meta/recipes-support/isar-cfg-rootpw/files/postinst.tmpl
> delete mode 100644 meta/recipes-support/isar-cfg-rootpw/isar-cfg-rootpw.bb
>
--
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 0/6] pre-processing pipeline and transient package replacement
2019-04-22 14:09 ` [PATCH 0/6] pre-processing pipeline and transient package replacement Maxim Yu. Osipov
@ 2019-04-24 7:11 ` Claudius Heine
2019-04-24 8:18 ` Maxim Yu. Osipov
0 siblings, 1 reply; 14+ messages in thread
From: Claudius Heine @ 2019-04-24 7:11 UTC (permalink / raw)
To: Maxim Yu. Osipov, isar-users; +Cc: Claudius Heine
Hi Maxim,
On 22/04/2019 16.09, Maxim Yu. Osipov wrote:
> The patch set doesn't apply against current 'next' (it would be more
> convenient to mention that this patchset depends on your previous series
> [PATCH v2 0/8] Cleanup rootfs creation).
I did in v1 of that patchset:
> this patchset contains some patches that where developed while
> implementing the preprocessing image pipeline. They are universally
> useful, but do prepare for the next steps.
But you are correct that I did not repeat myself in this one.
Maybe we should start looking into how to improve this process wise.
Would it help to post a git url for each patchset?
Maybe it might be possible to merge bigger patchset partially, so each
commit that looks good, instead of rejecting always whole patchsets?
The reason why I split patchsets up is that I hope that those will be
merged faster and thus lowering the work to constantly rebase everything
on the current next. So for example if patch 5 of 8 makes a problem,
just merge 1-4 and let the dev resent the fixed patchset containing just
patch 5 to 8. If that would be done then I would have no need to try
split patchsets up myself.
Claudius
>
> Maxim.
>
> On 4/18/19 1:25 PM, claudius.heine.ext@siemens.com wrote:
>> From: Claudius Heine <ch@denx.de>
>>
>> Hi,
>>
>> here is the tested, documented and slightly modified version of the
>> previous
>> RFC patch set.
>>
>> The CI has shown that the RFC patchset was not rebuildable. This
>> was patched by consolidating the `do_rootfs_prepare`,
>> `do_roofs_configure`
>> and `do_rootfs_install` into just one task `do_rootfs_install` that
>> now calls
>> the functions of both previous tasks as well as the `rootfs_prepare`
>> function
>> that previously was its own task `do_rootfs_prepare`.
>>
>> Otherwise I added documentation and more verbose commit messages.
>>
>> regards,
>> Claudius
>>
>> Claudius Heine (6):
>> split up isar-bootstrap helper and implement pre-process pipeline
>> meta: remove transient package support
>> meta/classes: add image-locales-extension class
>> meta/classes: add image-account-extension class
>> doc: update description of image customization
>> doc: some fixes
>>
>> doc/user_manual.md | 52 +++-
>> meta/classes/image-account-extension.bbclass | 253 ++++++++++++++++++
>> meta/classes/image-locales-extension.bbclass | 97 +++++++
>> meta/classes/image.bbclass | 30 +--
>> meta/classes/isar-bootstrap-helper.bbclass | 143 ----------
>> meta/classes/rootfs.bbclass | 176 +++++++++++-
>> .../isar-bootstrap/isar-bootstrap.inc | 5 +
>> .../buildchroot/buildchroot-host.bb | 16 +-
>> .../buildchroot/buildchroot-target.bb | 2 -
>> .../buildchroot/buildchroot.inc | 26 +-
>> meta/recipes-devtools/sdkchroot/sdkchroot.bb | 47 ++--
>> .../isar-cfg-localepurge/files/postinst | 15 --
>> .../isar-cfg-localepurge.bb | 70 -----
>> .../isar-cfg-rootpw/files/postinst.tmpl | 19 --
>> .../isar-cfg-rootpw/isar-cfg-rootpw.bb | 19 --
>> 15 files changed, 624 insertions(+), 346 deletions(-)
>> create mode 100644 meta/classes/image-account-extension.bbclass
>> create mode 100644 meta/classes/image-locales-extension.bbclass
>> delete mode 100644 meta/classes/isar-bootstrap-helper.bbclass
>> delete mode 100644
>> meta/recipes-support/isar-cfg-localepurge/files/postinst
>> delete mode 100644
>> meta/recipes-support/isar-cfg-localepurge/isar-cfg-localepurge.bb
>> delete mode 100644
>> meta/recipes-support/isar-cfg-rootpw/files/postinst.tmpl
>> delete mode 100644
>> meta/recipes-support/isar-cfg-rootpw/isar-cfg-rootpw.bb
>>
>
>
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 0/6] pre-processing pipeline and transient package replacement
2019-04-24 7:11 ` Claudius Heine
@ 2019-04-24 8:18 ` Maxim Yu. Osipov
2019-04-24 8:38 ` Claudius Heine
0 siblings, 1 reply; 14+ messages in thread
From: Maxim Yu. Osipov @ 2019-04-24 8:18 UTC (permalink / raw)
To: Claudius Heine, isar-users; +Cc: Claudius Heine
On 4/24/19 9:11 AM, Claudius Heine wrote:
> Hi Maxim,
>
> On 22/04/2019 16.09, Maxim Yu. Osipov wrote:
>> The patch set doesn't apply against current 'next' (it would be more
>> convenient to mention that this patchset depends on your previous
>> series [PATCH v2 0/8] Cleanup rootfs creation).
>
> I did in v1 of that patchset:
>
> > this patchset contains some patches that where developed while
> > implementing the preprocessing image pipeline. They are universally
> > useful, but do prepare for the next steps.
>
> But you are correct that I did not repeat myself in this one.
>
> Maybe we should start looking into how to improve this process wise.
>
> Would it help to post a git url for each patchset?
>
> Maybe it might be possible to merge bigger patchset partially, so each
> commit that looks good, instead of rejecting always whole patchsets?
>
> The reason why I split patchsets up is that I hope that those will be
> merged faster and thus lowering the work to constantly rebase everything
> on the current next. So for example if patch 5 of 8 makes a problem,
> just merge 1-4 and let the dev resent the fixed patchset containing just
> patch 5 to 8. If that would be done then I would have no need to try
> split patchsets up myself.
I agree that sometimes rejecting the whole patch set is not wise - but
it's up to patch series author to decide how to split patch sets.
Our policy states:
https://github.com/ilbers/isar/blob/master/CONTRIBUTING.md:
> 3. Every patch should implement only one logical modification. The
patch granularity is up to the developer. In general, smaller patches
with clear description are easier to review and accept.
>
> 4. Please provide patches that logically belong together in a series.
And vice-versa, please do not submit unrelated patches as series.
>
> Every series should have a cover letter with brief information about:
>
> What this series does.
>
> How it was tested.
>
> Diffstat (git format-patch --cover-letter does this for you).
>
Definitely, I'll test the whole series and I'll delegate responsibility
of fixing problems (if found during testing) in patch set to the author.
Maxim.
> Claudius
>
>>
>> Maxim.
>>
>> On 4/18/19 1:25 PM, claudius.heine.ext@siemens.com wrote:
>>> From: Claudius Heine <ch@denx.de>
>>>
>>> Hi,
>>>
>>> here is the tested, documented and slightly modified version of the
>>> previous
>>> RFC patch set.
>>>
>>> The CI has shown that the RFC patchset was not rebuildable. This
>>> was patched by consolidating the `do_rootfs_prepare`,
>>> `do_roofs_configure`
>>> and `do_rootfs_install` into just one task `do_rootfs_install` that
>>> now calls
>>> the functions of both previous tasks as well as the `rootfs_prepare`
>>> function
>>> that previously was its own task `do_rootfs_prepare`.
>>>
>>> Otherwise I added documentation and more verbose commit messages.
>>>
>>> regards,
>>> Claudius
>>>
>>> Claudius Heine (6):
>>> split up isar-bootstrap helper and implement pre-process pipeline
>>> meta: remove transient package support
>>> meta/classes: add image-locales-extension class
>>> meta/classes: add image-account-extension class
>>> doc: update description of image customization
>>> doc: some fixes
>>>
>>> doc/user_manual.md | 52 +++-
>>> meta/classes/image-account-extension.bbclass | 253 ++++++++++++++++++
>>> meta/classes/image-locales-extension.bbclass | 97 +++++++
>>> meta/classes/image.bbclass | 30 +--
>>> meta/classes/isar-bootstrap-helper.bbclass | 143 ----------
>>> meta/classes/rootfs.bbclass | 176 +++++++++++-
>>> .../isar-bootstrap/isar-bootstrap.inc | 5 +
>>> .../buildchroot/buildchroot-host.bb | 16 +-
>>> .../buildchroot/buildchroot-target.bb | 2 -
>>> .../buildchroot/buildchroot.inc | 26 +-
>>> meta/recipes-devtools/sdkchroot/sdkchroot.bb | 47 ++--
>>> .../isar-cfg-localepurge/files/postinst | 15 --
>>> .../isar-cfg-localepurge.bb | 70 -----
>>> .../isar-cfg-rootpw/files/postinst.tmpl | 19 --
>>> .../isar-cfg-rootpw/isar-cfg-rootpw.bb | 19 --
>>> 15 files changed, 624 insertions(+), 346 deletions(-)
>>> create mode 100644 meta/classes/image-account-extension.bbclass
>>> create mode 100644 meta/classes/image-locales-extension.bbclass
>>> delete mode 100644 meta/classes/isar-bootstrap-helper.bbclass
>>> delete mode 100644
>>> meta/recipes-support/isar-cfg-localepurge/files/postinst
>>> delete mode 100644
>>> meta/recipes-support/isar-cfg-localepurge/isar-cfg-localepurge.bb
>>> delete mode 100644
>>> meta/recipes-support/isar-cfg-rootpw/files/postinst.tmpl
>>> delete mode 100644
>>> meta/recipes-support/isar-cfg-rootpw/isar-cfg-rootpw.bb
>>>
>>
>>
>
--
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 0/6] pre-processing pipeline and transient package replacement
2019-04-24 8:18 ` Maxim Yu. Osipov
@ 2019-04-24 8:38 ` Claudius Heine
2019-04-24 14:38 ` Baurzhan Ismagulov
0 siblings, 1 reply; 14+ messages in thread
From: Claudius Heine @ 2019-04-24 8:38 UTC (permalink / raw)
To: Maxim Yu. Osipov, isar-users; +Cc: Claudius Heine
On 24/04/2019 10.18, Maxim Yu. Osipov wrote:
> On 4/24/19 9:11 AM, Claudius Heine wrote:
>> Hi Maxim,
>>
>> On 22/04/2019 16.09, Maxim Yu. Osipov wrote:
>>> The patch set doesn't apply against current 'next' (it would be more
>>> convenient to mention that this patchset depends on your previous
>>> series [PATCH v2 0/8] Cleanup rootfs creation).
>>
>> I did in v1 of that patchset:
>>
>> > this patchset contains some patches that where developed while
>> > implementing the preprocessing image pipeline. They are universally
>> > useful, but do prepare for the next steps.
>>
>> But you are correct that I did not repeat myself in this one.
>>
>> Maybe we should start looking into how to improve this process wise.
>>
>> Would it help to post a git url for each patchset?
>>
>> Maybe it might be possible to merge bigger patchset partially, so each
>> commit that looks good, instead of rejecting always whole patchsets?
>
>>
>> The reason why I split patchsets up is that I hope that those will be
>> merged faster and thus lowering the work to constantly rebase
>> everything on the current next. So for example if patch 5 of 8 makes a
>> problem, just merge 1-4 and let the dev resent the fixed patchset
>> containing just patch 5 to 8. If that would be done then I would have
>> no need to try split patchsets up myself.
>
> I agree that sometimes rejecting the whole patch set is not wise - but
> it's up to patch series author to decide how to split patch sets.
>
> Our policy states:
>
> https://github.com/ilbers/isar/blob/master/CONTRIBUTING.md:
>
> > 3. Every patch should implement only one logical modification. The
> patch granularity is up to the developer. In general, smaller patches
> with clear description are easier to review and accept.
> >
> > 4. Please provide patches that logically belong together in a series.
> And vice-versa, please do not submit unrelated patches as series.
> >
> > Every series should have a cover letter with brief information about:
> >
> > What this series does.
> >
> > How it was tested.
> >
> > Diffstat (git format-patch --cover-letter does this for you).
> >
>
> Definitely, I'll test the whole series and I'll delegate responsibility
> of fixing problems (if found during testing) in patch set to the author.
Well there are different kind of patchsets, some patchsets implement
features and some just cleanup/refactor code. The latter ones might
contain a number of small improvements all over the code base. They are
related in a sense that they 'do' the same (refactoring/cleanup), not
necessarily with the same code pieces. But they might work towards
cleaning up/refactoring/preparing the code for a bigger feature or
rework that comes later.
So what you are saying that you would prefer instead of putting all
those small refactoring changes together in one bigger patchset, that I
should submit all those patches separate from each other? An then state
that the feature/rework patchset depends on all of them?
I think that will cause more work on every ones end (especially with the
slow CI build), but if that is what needs to be done when contributing
to Isar, I will look into reordering my patchsets accordingly and maybe
write some scripts to automate this pipeline for myself.
Claudius
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 0/6] pre-processing pipeline and transient package replacement
2019-04-24 8:38 ` Claudius Heine
@ 2019-04-24 14:38 ` Baurzhan Ismagulov
2019-04-24 14:55 ` Claudius Heine
0 siblings, 1 reply; 14+ messages in thread
From: Baurzhan Ismagulov @ 2019-04-24 14:38 UTC (permalink / raw)
To: isar-users
On Wed, Apr 24, 2019 at 10:38:08AM +0200, Claudius Heine wrote:
> Well there are different kind of patchsets, some patchsets implement
> features and some just cleanup/refactor code. The latter ones might contain
> a number of small improvements all over the code base. They are related in a
> sense that they 'do' the same (refactoring/cleanup), not necessarily with
> the same code pieces. But they might work towards cleaning
> up/refactoring/preparing the code for a bigger feature or rework that comes
> later.
>
> So what you are saying that you would prefer instead of putting all those
> small refactoring changes together in one bigger patchset, that I should
> submit all those patches separate from each other? An then state that the
> feature/rework patchset depends on all of them?
>
> I think that will cause more work on every ones end (especially with the
> slow CI build), but if that is what needs to be done when contributing to
> Isar, I will look into reordering my patchsets accordingly and maybe write
> some scripts to automate this pipeline for myself.
The current practice resulted from past discussions, in which amending patches
without reviewing again or applying series in part had been objected to. I
haven't looked at the proposals in detail yet, but in general, if we can
improve the process to save overall effort, I'm for it.
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 0/6] pre-processing pipeline and transient package replacement
2019-04-24 14:38 ` Baurzhan Ismagulov
@ 2019-04-24 14:55 ` Claudius Heine
2019-04-25 10:06 ` Baurzhan Ismagulov
0 siblings, 1 reply; 14+ messages in thread
From: Claudius Heine @ 2019-04-24 14:55 UTC (permalink / raw)
To: isar-users
Hi Baurzahn,
On 24/04/2019 16.38, Baurzhan Ismagulov wrote:
> On Wed, Apr 24, 2019 at 10:38:08AM +0200, Claudius Heine wrote:
>> Well there are different kind of patchsets, some patchsets implement
>> features and some just cleanup/refactor code. The latter ones might contain
>> a number of small improvements all over the code base. They are related in a
>> sense that they 'do' the same (refactoring/cleanup), not necessarily with
>> the same code pieces. But they might work towards cleaning
>> up/refactoring/preparing the code for a bigger feature or rework that comes
>> later.
>>
>> So what you are saying that you would prefer instead of putting all those
>> small refactoring changes together in one bigger patchset, that I should
>> submit all those patches separate from each other? An then state that the
>> feature/rework patchset depends on all of them?
>>
>> I think that will cause more work on every ones end (especially with the
>> slow CI build), but if that is what needs to be done when contributing to
>> Isar, I will look into reordering my patchsets accordingly and maybe write
>> some scripts to automate this pipeline for myself.
>
> The current practice resulted from past discussions, in which amending patches
> without reviewing again or applying series in part had been objected to.
IIRC there were objections about reordering patches within one series,
since they normally depend on each other and always describing those
dependencies explicitly does not really make sense IMO. I would also
object to modifying a patch content without making it clear first what
the issue with it is.
Fixing spelling or grammar mistakes is fine IMO, especially in commit
messages, as long as it doesn't change the content itself to much.
Just applying some cleanup and refactoring patches from a series in
order, without applying the feature (if it contains one) or some other
latter problematic patches is fine as well IMO.
> I
> haven't looked at the proposals in detail yet, but in general, if we can
> improve the process to save overall effort, I'm for it.
Well the first goal should be to improve the code. Therefore fixes,
cleanup and refactoring patches, if they make sense, should be merged
fast. For features, new dependencies or bigger/backward incompatible
rework (UX changes), there should be a discussion etc. maybe with help
of an RFC patch to demonstrate those ideas.
Claudius
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 0/6] pre-processing pipeline and transient package replacement
2019-04-24 14:55 ` Claudius Heine
@ 2019-04-25 10:06 ` Baurzhan Ismagulov
0 siblings, 0 replies; 14+ messages in thread
From: Baurzhan Ismagulov @ 2019-04-25 10:06 UTC (permalink / raw)
To: isar-users
On Wed, Apr 24, 2019 at 04:55:36PM +0200, Claudius Heine wrote:
> Just applying some cleanup and refactoring patches from a series in order,
> without applying the feature (if it contains one) or some other latter
> problematic patches is fine as well IMO.
If you are aware which patches do and which don't cause the problems and would
like to get them applied faster, one option is to resend the partial working
series first and post the rest after sorting out the issues.
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2019-04-25 10:06 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-04-18 11:25 [PATCH 0/6] pre-processing pipeline and transient package replacement claudius.heine.ext
2019-04-18 11:25 ` [PATCH 1/6] split up isar-bootstrap helper and implement pre-process pipeline claudius.heine.ext
2019-04-18 11:25 ` [PATCH 2/6] meta: remove transient package support claudius.heine.ext
2019-04-18 11:25 ` [PATCH 3/6] meta/classes: add image-locales-extension class claudius.heine.ext
2019-04-18 11:25 ` [PATCH 4/6] meta/classes: add image-account-extension class claudius.heine.ext
2019-04-18 11:25 ` [PATCH 5/6] doc: update description of image customization claudius.heine.ext
2019-04-18 11:25 ` [PATCH 6/6] doc: some fixes claudius.heine.ext
2019-04-22 14:09 ` [PATCH 0/6] pre-processing pipeline and transient package replacement Maxim Yu. Osipov
2019-04-24 7:11 ` Claudius Heine
2019-04-24 8:18 ` Maxim Yu. Osipov
2019-04-24 8:38 ` Claudius Heine
2019-04-24 14:38 ` Baurzhan Ismagulov
2019-04-24 14:55 ` Claudius Heine
2019-04-25 10:06 ` Baurzhan Ismagulov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox