[PATCH v6 0/4] One-shot configuration packages

[PATCH v6 0/4] One-shot configuration packages

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

Hi,

I decided to merge the isar-cfg-localpurge improvement patch series with
this one, since they touch the same code basis.

With this change the isar-cfg-localepurge package code is pretty much
done from stretch. The configuration now depends on bitbake internal
variables, so now its possible to set the desired locale without need to
overwrite any recipes, just by setting the `LOCALE_GEN` and
`LOCALE_DEFAULT` variables.

I also removed the debconf configuration of localepurge. This part is
pretty broken in that package. Now '/etc/locale.nopurge' is just
generated and placed by bitbake and the postinst script.

Claudius

Changes from v5:
 - reworked isar-cfg-localepurge package

Changes from v4:
 - added documentation
 - renamed `IMAGE_CFG_PACKAGE` to `IMAGE_TRANSIENT_PACKAGES`

Changes from v3:
 - Purge one-shot packages in reverse order

Changes from v2:
 - Install one-shot package with a loop
 - Rename '-y' apt parameter to '--yes'
 - Move isar-cfg-localepurge files from /usr/local/etc to /usr/lib/

Changes from v1:
 - rebased to current next
 - removed locale.nopurge setup in isar-bootstrap

Claudius Heine (4):
  isar-bootstrap|configscript: implement one transient packages
  meta/isar-cfg-localepurge: added locale purge recipe
  meta-isar/isar-image-base: switch use isar-cfg-localepurge
  doc: added description of `IMAGE_TRANSIENT_PACKAGES` and locale vars

 doc/user_manual.md                            | 18 +++++
 .../images/files/debian-configscript.sh       |  9 ---
 .../images/files/raspbian-configscript.sh     |  8 ---
 .../recipes-core/images/isar-image-base.bb    |  6 +-
 meta/classes/isar-bootstrap-helper.bbclass    | 32 +++++++--
 .../isar-bootstrap/files/locale.nopurge       | 10 ---
 .../isar-bootstrap/isar-bootstrap.bb          |  4 +-
 .../isar-cfg-localepurge/files/postinst       | 13 ++++
 .../isar-cfg-localepurge.bb                   | 65 +++++++++++++++++++
 9 files changed, 128 insertions(+), 37 deletions(-)
 delete mode 100644 meta/recipes-core/isar-bootstrap/files/locale.nopurge
 create mode 100644 meta/recipes-support/isar-cfg-localepurge/files/postinst
 create mode 100644 meta/recipes-support/isar-cfg-localepurge/isar-cfg-localepurge.bb

-- 
2.17.0

[PATCH v6 1/4] isar-bootstrap|configscript: implement one transient packages

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

Some configuration tasks on the image require certain other packages,
but those packages might not be required on the final image.

One way to solve this is use a special package that is installed to the
image and then removed again. When installing it makes certain that all
required packages are installed to the image as well. Then the image
can be configured in the postinst hook. When this package is removed and
all unused dependencies are removed as well, no not required packages
used by this package should remain on the image.

Signed-off-by: Claudius Heine <ch@denx.de>
---
 .../images/files/debian-configscript.sh       |  1 -
 .../images/files/raspbian-configscript.sh     |  1 -
 meta/classes/isar-bootstrap-helper.bbclass    | 32 ++++++++++++++++---
 3 files changed, 28 insertions(+), 6 deletions(-)

diff --git a/meta-isar/recipes-core/images/files/debian-configscript.sh b/meta-isar/recipes-core/images/files/debian-configscript.sh
index 7ca0562..9b3ff30 100755
--- a/meta-isar/recipes-core/images/files/debian-configscript.sh
+++ b/meta-isar/recipes-core/images/files/debian-configscript.sh
@@ -40,4 +40,3 @@ fi
 
 # Purge unused locale and installed packages' .deb files
 localepurge
-apt-get clean
diff --git a/meta-isar/recipes-core/images/files/raspbian-configscript.sh b/meta-isar/recipes-core/images/files/raspbian-configscript.sh
index b240de9..448dea5 100644
--- a/meta-isar/recipes-core/images/files/raspbian-configscript.sh
+++ b/meta-isar/recipes-core/images/files/raspbian-configscript.sh
@@ -40,4 +40,3 @@ kernel=$KERNEL_IMAGE
 EOF
 
 localepurge
-apt-get clean
diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index 4284d02..d6b2a26 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass
@@ -5,15 +5,24 @@
 #
 # SPDX-License-Identifier: MIT
 
+IMAGE_TRANSIENT_PACKAGES ??= ""
+
+def reverse_bb_array(d, varname):
+    array = d.getVar(varname, True)
+    if array is None:
+        return None
+    array = reversed(array.split())
+    return " ".join(i for i in array)
+
 setup_root_file_system() {
     ROOTFSDIR="$1"
-    CLEANHOSTLEAK="$2"
+    CLEAN="$2"
     shift
     shift
     PACKAGES="$@"
     APT_ARGS="install --yes --allow-unauthenticated \
               -o Debug::pkgProblemResolver=yes"
-    CLEANHOSTLEAK_FILES="${ROOTFSDIR}/etc/hostname ${ROOTFSDIR}/etc/resolv.conf"
+    CLEAN_FILES="${ROOTFSDIR}/etc/hostname ${ROOTFSDIR}/etc/resolv.conf"
 
     sudo cp -Trpfx \
         "${DEPLOY_DIR_IMAGE}/isar-bootstrap-${DISTRO}-${DISTRO_ARCH}/" \
@@ -41,8 +50,23 @@ setup_root_file_system() {
         -o Dir::Etc::sourceparts="-" \
         -o APT::Get::List-Cleanup="0"
     sudo -E chroot "$ROOTFSDIR" \
-        /usr/bin/apt-get ${APT_ARGS} --download-only $PACKAGES
-    [ "clean" = ${CLEANHOSTLEAK} ] && sudo rm -f ${CLEANHOSTLEAK_FILES}
+        /usr/bin/apt-get ${APT_ARGS} --download-only $PACKAGES \
+            ${IMAGE_TRANSIENT_PACKAGES}
+    [ "clean" = ${CLEAN} ] && sudo rm -f ${CLEAN_FILES}
     sudo -E chroot "$ROOTFSDIR" \
         /usr/bin/apt-get ${APT_ARGS} $PACKAGES
+    for pkg in ${IMAGE_TRANSIENT_PACKAGES}; do
+        sudo -E chroot "$ROOTFSDIR" \
+            /usr/bin/apt-get ${APT_ARGS} $pkg
+    done
+    for pkg in ${@reverse_bb_array(d, "IMAGE_TRANSIENT_PACKAGES") or ""}; do
+        sudo -E chroot "$ROOTFSDIR" \
+            /usr/bin/apt-get purge --yes $pkg
+    done
+    if [ "clean" = ${CLEAN} ]; then
+        sudo -E chroot "$ROOTFSDIR" \
+            /usr/bin/apt-get autoremove --purge --yes
+        sudo -E chroot "$ROOTFSDIR" \
+            /usr/bin/apt-get clean
+    fi
 }
-- 
2.17.0

[PATCH v6 2/4] meta/isar-cfg-localepurge: added locale purge recipe

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

The package 'isar-cfg-localepurge' is used to set the locales of the
root file system and purge all not required locales.

The variables used to configure the locale with this package are `LOCALE_GEN`
which is basically the content of the desired `/etc/locale.gen` with each
line seperated by '\n' and `LOCALE_DEFAULT` which sets the global default
locale of the system.

Signed-off-by: Claudius Heine <ch@denx.de>
---
 .../isar-cfg-localepurge/files/postinst       | 13 ++++
 .../isar-cfg-localepurge.bb                   | 65 +++++++++++++++++++
 2 files changed, 78 insertions(+)
 create mode 100644 meta/recipes-support/isar-cfg-localepurge/files/postinst
 create mode 100644 meta/recipes-support/isar-cfg-localepurge/isar-cfg-localepurge.bb

diff --git a/meta/recipes-support/isar-cfg-localepurge/files/postinst b/meta/recipes-support/isar-cfg-localepurge/files/postinst
new file mode 100644
index 0000000..c5f1c51
--- /dev/null
+++ b/meta/recipes-support/isar-cfg-localepurge/files/postinst
@@ -0,0 +1,13 @@
+#!/bin/sh
+set -e
+
+cat /usr/lib/isar-cfg-localepurge/locale.gen \
+	>> /etc/locale.gen
+cat /usr/lib/isar-cfg-localepurge/locale.nopurge \
+	> /etc/locale.nopurge
+
+debconf-set-selections /usr/lib/isar-cfg-localepurge/locale.debconf
+
+dpkg-reconfigure -f noninteractive locales
+
+localepurge
diff --git a/meta/recipes-support/isar-cfg-localepurge/isar-cfg-localepurge.bb b/meta/recipes-support/isar-cfg-localepurge/isar-cfg-localepurge.bb
new file mode 100644
index 0000000..bdbc2b1
--- /dev/null
+++ b/meta/recipes-support/isar-cfg-localepurge/isar-cfg-localepurge.bb
@@ -0,0 +1,65 @@
+# This software is a part of ISAR.
+
+DESCRIPTION = "Isar configuration package for locale and localepurge"
+MAINTAINER = "isar-users <isar-users@googlegroups.com>"
+DEBIAN_DEPENDS = "localepurge"
+
+SRC_URI = "file://postinst"
+
+inherit dpkg-raw
+
+LOCALE_GEN ?= "en_US.UTF-8 UTF-8\n\
+               en_US ISO-8859-1\n"
+LOCALE_DEFAULT ?= "en_US.UTF-8"
+
+def get_locale_gen(d):
+    locale_gen = d.getVar("LOCALE_GEN", True) or ""
+    return '\n'.join(sorted(set(i.strip()
+                                for i in locale_gen.split('\\n')
+                                if i.strip())))
+
+def get_dc_locale_gen(d):
+    locale_gen = d.getVar("LOCALE_GEN", True) or ""
+    return ', '.join(sorted(set(i.strip()
+                                for i in locale_gen.split('\\n')
+                                if i.strip())))
+
+def get_nopurge(d):
+    locale_gen = d.getVar("LOCALE_GEN", True) or ""
+    return '\n'.join(sorted(set(i.strip()
+                                for j in locale_gen.split('\\n')
+                                if j.strip()
+                                for i in (j.split()[0].split("_")[0],
+                                          j.split()[0].split(".")[0],
+                                          j.split()[0]))))
+
+do_gen_config[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+do_gen_config() {
+	cat<<-__EOF__ > ${WORKDIR}/locale.gen
+		${@get_locale_gen(d)}
+	__EOF__
+	cat<<-__EOF__ > ${WORKDIR}/locale.debconf
+		locales     locales/locales_to_be_generated    multiselect ${@get_dc_locale_gen(d)}
+		locales     locales/default_environment_locale select      ${LOCALE_DEFAULT}
+	__EOF__
+	cat<<-__EOF__ > ${WORKDIR}/locale.nopurge
+		#USE_DPKG
+		MANDELETE
+		DONTBOTHERNEWLOCALE
+		#SHOWFREEDSPACE
+		#QUICKNDIRTYCALC
+		#VERBOSE
+		${@get_nopurge(d)}
+	__EOF__
+}
+addtask gen_config after do_unpack before do_install
+
+do_install() {
+	install -v -d ${D}/usr/lib/${PN}
+	install -v -m 644 ${WORKDIR}/locale.debconf \
+                          ${D}/usr/lib/${PN}/locale.debconf
+	install -v -m 644 ${WORKDIR}/locale.gen \
+                          ${D}/usr/lib/${PN}/locale.gen
+	install -v -m 644 ${WORKDIR}/locale.nopurge \
+                          ${D}/usr/lib/${PN}/locale.nopurge
+}
-- 
2.17.0

[PATCH v6 3/4] meta-isar/isar-image-base: switch use isar-cfg-localepurge

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

This patch removes the locale setting from the *-configscripts and
instead uses the isar-cfg-localepurge configuration package as a
transient package.

The advantange is that the package 'localepurge' is no longer installed
on every image and locale settings is done in a central place.

Signed-off-by: Claudius Heine <ch@denx.de>
---
 .../recipes-core/images/files/debian-configscript.sh   |  8 --------
 .../recipes-core/images/files/raspbian-configscript.sh |  7 -------
 meta-isar/recipes-core/images/isar-image-base.bb       |  6 +++---
 meta/recipes-core/isar-bootstrap/files/locale.nopurge  | 10 ----------
 meta/recipes-core/isar-bootstrap/isar-bootstrap.bb     |  4 +---
 5 files changed, 4 insertions(+), 31 deletions(-)
 delete mode 100644 meta/recipes-core/isar-bootstrap/files/locale.nopurge

diff --git a/meta-isar/recipes-core/images/files/debian-configscript.sh b/meta-isar/recipes-core/images/files/debian-configscript.sh
index 9b3ff30..136918c 100755
--- a/meta-isar/recipes-core/images/files/debian-configscript.sh
+++ b/meta-isar/recipes-core/images/files/debian-configscript.sh
@@ -10,11 +10,6 @@ readonly BAUDRATE_TTY="$2"
 readonly ROOTFS_DEV="$3"
 readonly ROOTFS_TYPE="$4"
 
-debconf-set-selections <<END
-locales locales/locales_to_be_generated multiselect en_US.UTF-8 UTF-8
-locales locales/default_environment_locale select en_US.UTF-8
-END
-
 cat > /etc/fstab << EOF
 # Begin /etc/fstab
 /dev/$ROOTFS_DEV	/		$ROOTFS_TYPE		defaults		1	1
@@ -37,6 +32,3 @@ if [ -f /etc/inittab ]; then
     echo "T0:23:respawn:/sbin/getty -L $MACHINE_SERIAL $BAUDRATE_TTY vt100" \
         >> /etc/inittab
 fi
-
-# Purge unused locale and installed packages' .deb files
-localepurge
diff --git a/meta-isar/recipes-core/images/files/raspbian-configscript.sh b/meta-isar/recipes-core/images/files/raspbian-configscript.sh
index 448dea5..04ea571 100644
--- a/meta-isar/recipes-core/images/files/raspbian-configscript.sh
+++ b/meta-isar/recipes-core/images/files/raspbian-configscript.sh
@@ -10,11 +10,6 @@ readonly BAUDRATE_TTY="$2"
 readonly ROOTFS_DEV="$3"
 readonly ROOTFS_TYPE="$4"
 
-debconf-set-selections <<END
-locales locales/locales_to_be_generated multiselect en_US.UTF-8 UTF-8
-locales locales/default_environment_locale select en_US.UTF-8
-END
-
 cat > /etc/fstab << EOF
 # Begin /etc/fstab
 /dev/$ROOTFS_DEV	/		$ROOTFS_TYPE		defaults		1	1
@@ -38,5 +33,3 @@ KERNEL_IMAGE=`ls /boot | grep vmlinuz`
 cat > /boot/config.txt << EOF
 kernel=$KERNEL_IMAGE
 EOF
-
-localepurge
diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index 1239f72..f91dfb0 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -16,11 +16,11 @@ PV = "1.0"
 inherit image
 inherit isar-bootstrap-helper
 
-DEPENDS += "${IMAGE_INSTALL}"
+DEPENDS += "${IMAGE_INSTALL} ${IMAGE_TRANSIENT_PACKAGES}"
 
 IMAGE_PREINSTALL += "apt \
-                     dbus \
-                     localepurge"
+                     dbus"
+IMAGE_TRANSIENT_PACKAGES += "isar-cfg-localepurge"
 
 WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
 
diff --git a/meta/recipes-core/isar-bootstrap/files/locale.nopurge b/meta/recipes-core/isar-bootstrap/files/locale.nopurge
deleted file mode 100644
index 510d40c..0000000
--- a/meta/recipes-core/isar-bootstrap/files/locale.nopurge
+++ /dev/null
@@ -1,10 +0,0 @@
-# Remove localized man pages
-MANDELETE
-
-# Delete new locales which appear on the system without bothering you
-DONTBOTHERNEWLOCALE
-
-# Keep these locales after package installations via apt-get(8)
-en
-en_US
-en_US.UTF-8
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb
index f894821..bb3992b 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb
@@ -13,8 +13,7 @@ FILESPATH_prepend := "${THISDIR}/files:"
 SRC_URI = " \
     file://isar-apt.conf \
     file://isar-apt-fallback.conf \
-    file://locale \
-    file://locale.nopurge"
+    file://locale"
 PV = "1.0"
 
 WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
@@ -196,7 +195,6 @@ addtask bootstrap before do_build after do_generate_keyring
 
 do_set_locale() {
     sudo install -v -m644 "${WORKDIR}/locale" "${ROOTFSDIR}/etc/locale"
-    sudo install -v -m644 "${WORKDIR}/locale.nopurge" "${ROOTFSDIR}/etc/locale.nopurge"
 
     sudo sed -i '/en_US.UTF-8 UTF-8/s/^#//g' "${ROOTFSDIR}/etc/locale.gen"
     sudo -E chroot "${ROOTFSDIR}" /usr/sbin/locale-gen
-- 
2.17.0

[PATCH v6 4/4] doc: added description of `IMAGE_TRANSIENT_PACKAGES` and locale vars

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

---
 doc/user_manual.md | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index a2f4b92..9921f3a 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -318,6 +318,8 @@ Some other variables include:
 
  - `IMAGE_INSTALL` - The list of custom packages to build and install to target image, please refer to relative chapter for more information.
  - `BB_NUMBER_THREADS` - The number of `bitbake` jobs that can be run in parallel. Please set this option according your host CPU cores number.
+ - `LOCALE_GEN` - A `\n` seperated list of `/etc/locale.gen` entries desired on the target.
+ - `LOCALE_DEFAULT` - The default locale used for the `LANG` and `LANGUAGE` variable in `/etc/locale`.
 
 ---
 
@@ -481,6 +483,22 @@ Isar contains two image type classes that can be used as reference:
 
 ---
 
+## Customize and configure image
+
+Customization and configuration of an image should be done via packages, see below.
+
+Adding those configuration packages to the image can be done in two ways:
+
+ 1. Simply adding the package to `IMAGE_INSTALL`, like any other isar created package, or
+ 2. Adding the package to `IMAGE_TRANSIENT_PACKAGES`.
+
+In most cases adding the configuration package to `IMAGE_INSTALL` is the right option.
+
+In cases were the configuration script of the package has some external dependencies, that should not be part of the final image, then `IMAGE_TRANSIENT_PACKAGES` is the right option.
+Packages in the `IMAGE_TRANSIENT_PACKAGES` variable are installed to the image and purged in the next step. If such a configuration package deploys file as part of their content, then those files will be removed as well.
+
+---
+
 ## Add a Custom Application
 
 Before creating a new recipe it's highly recommended to take a look into the BitBake user manual mentioned in Terms and Definitions section.
-- 
2.17.0