Re: [PATCHSET] expand last partition script and its usage

["Roberto A. Foglietta" <roberto.foglietta@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 2275 bytes --]

Hi all,

this is the single patch for rework and generalisation of last
partition expansion.
As usual has been sent by elastic mail SMTP and it went in moderation.

UPDATE: the unpartition volume resizing has been tested in this way

git clone https://github.com/robang74/isar-nvidia-debian.git
cd isar-nvidia-debian
source .profile
build basic-os
wicshell
        cd /boot
        cp -arf vmlinuz-5.10.0-19-amd64 initrd.img-5.10.0-19-amd64 /work
        exit
wicinst file:image.wic
sudo losetup -P $(losetup -f) image.wic
sudo dd if=/dev/loop3p2 bs=1M > disk.wic
dd if=/dev/zero bs=1 seek=$[4*1024*1024*1024] count=1 oflag=append
conv=notrunc,sparse status=none of=disk.wic
sudo qemu-system-x86_64 --cpu host --enable-kvm -smp 4 -m 1024 -drive
format=raw,file=disk.wic -device sdhci-pci -kernel
build/tmp/work/vmlinuz-5.10.0-19-amd64 -initrd
build/tmp/work/initrd.img-5.10.0-19-amd64 -append "root=/dev/sda
console=ttyS0" --nographic; echo ">>> press ENTER"; read; reset
       dmesg | grep expand
       df -h
       poweroff

Best regards, R-

===========================================

The script has been reworked to simplify its code:

 - the last partition free space is not computed anymore because useless
 - on exit a function take care of some final tasks and logs in /dev/kmsg

and to extend its functionalities supporting new filesystems:

 - filesystem type is managed by a switch-case block of code
 - temporary directory to mount is created relying on TMPDIR
 - unsupervised systems have the option to reboot because error
 - if temporary directory creation fails 2nd try with /dev/shm
 - additional ext2, ext3 and btrfs filesystems supported
 - udev support: udevadm settle is the first task to run
 - unpartitioned volumes are resized as well (tested)

This patch has been created by a collection of 7 patches to be applied on
ilbers ISAR, branch next, commit d26660b724b034b602f3889f55a23cd9be2e87bd

   Author: Roberto A. Foglietta <roberto.foglietta@gmail.com>

This code has been developed taking inspiration by suggestions and code from:

   Felix Moessbauer <felix.moessbauer@siemens.com>
   Tobias Schmidl <tobiasschmidl@siemens.com>
   Henning Schild <henning.schild@siemens.com>

Signed-off-by: Roberto A. Foglietta <roberto.foglietta@gmail.com>

[-- Attachment #2: 0001-The-expand-last-partition-rework-and-generalisation.patch --]
[-- Type: text/x-patch, Size: 12317 bytes --]

From b41bef462289422e9fedc8d87706e11f267b2c37 Mon Sep 17 00:00:00 2001
From: "Roberto A. Foglietta" <roberto.foglietta@gmail.com>
Date: Mon, 12 Dec 2022 21:14:26 +0100
Subject: [PATCH] The expand last partition rework and generalisation

The script has been reowrked to simplify its code:

 - the last partition free space is not computed anymore because useless
 - on exit a function take care of some final tasks and logs in /dev/kmsg

and to extend its functionalities supporting new filesystems:

 - filesystem type is managed by a switch-case block of code
 - temporary directory to mount is created relying on TMPDIR
 - unsupervised systems have the option to reboot because error
 - if temporary directory creation fails 2nd try with /dev/shm
 - additional ext2, ext3 and btrfs filesystems supported
 - udev support: udevadm settle is the first task to run
 - unpartitioned volumes are resized as well (tested)

This patch has been created by a collection of 7 patches to be applied on
ilbers ISAR, branch next, commit d26660b724b034b602f3889f55a23cd9be2e87bd

   Author: Roberto A. Foglietta <roberto.foglietta@gmail.com>

This code has been developed taking inspiration by suggestions and code from:

   Felix Moessbauer <felix.moessbauer@siemens.com>
   Tobias Schmidl <tobiasschmidl@siemens.com>
   Henning Schild <henning.schild@siemens.com>

Signed-off-by: Roberto A. Foglietta <roberto.foglietta@gmail.com>

= 1/7 == In expand last partition wait for udev have finished =================

bugfix, expand last partition, wait for udev completion

On some hardware the udev did not create the links to the disk partitions
while this script is working thus it fails. This patch fixes those cases.

v2: as underlined by Tobias this script requires systemd-udevd.service and
it can safely run only after that service has been started otherwise it might
fail. This dependency will reorder the boot procedure but not slow down it
unless the hardware is slow to get mapped by udev but in that case resize
the last partition would have failed. Thus, no regression is expected.

= 2/7 == In expand last partition script btrfs support added ==================

Improvement, expand-on-first-boot: support for btrfs added

v2: the mount point for btrfs filesystem to resize is under /dev/shm
by default but a variable defined in the running environment can
change the temporary directory path. The /dev/shm has been chosen
because in some systems /tmp can be on a RO root filesystem and
/tmp not yet mounted with tmpfs. This requires the system to have
/dev/shm configured into the kernel.

v3: the previous version was using a bashism but the shebang is /bin/sh
this means that the script will fail in a customized system in which /bin/sh
do not link to bash or bash is not available.

v4: Felix suggested using mktemp for the temporary directory in which to
mount the btrfs filesystem to resize. The default path /tmp could be changed
using the variable TMPDIR to set in the running environment.

v5: typo fixes into the patch description

v6: mktemp does a reasonable second try with -p /dev/shm before failing

= 3/7 == If expand last partition fails, unsupervised systems reboot ==========

Improvement for unsupervised embedded devices which need to reboot if expanding
last partition fails because an under-provided system should not even complete
the boot in the field in order to avoid functioning failures at unexpected
future time. This patch do not affect the previous behaviour because requires
that /etc/unsupervised file exists.

v2: if there is no partition table exit 0 instead of exit 1

= 4/7 == In expand last partition checking the size is useless ================

code maintenance, superfluous operations commented, kept for the future

Tests shown that resize in full a ext4 or btrfs partition/filesystem does not
hurt and does not report a failure, so nothing changes without the size check

= 5/7 == The expand last partition might support full volumes also, to test ==

Improvement, expansion of unpartitioned volumes supported also but to test

DONE, 14.12.2022: this change should be tested with a single volume system
preliminary tests shown that ext4 and btrfs entire volumes cam be resized

= 6/7 == In expand last partition trap exitnlog EXIT logs with simpler code ===

Improvement, the script log at exit with the status of completion and its full
path in this way - in case of failure - the user will find in dmesg the error
and also the script full path to debug. It uses /dev/kmsg to log.
Code simplified.

= 7/7 == In expand last partition few improvements and a bugfix ===============

Some improvements and a bugfix in expanding last partition.

Improvements: filesystems ext2 and ext3 also supported, warning in dmesg in
case of unsupported filesystem, the variable tmpdir is used with double
quotes even if unneeded but be consistent with the original script notation.

Bugfix: even if btrfs resize fails, umount the last partition and return error.
---
 .../expand-on-first-boot_1.2.bb               |   2 +-
 .../files/expand-last-partition.sh            | 134 ++++++++++--------
 .../files/expand-on-first-boot.service        |   2 +
 3 files changed, 77 insertions(+), 61 deletions(-)

diff --git a/meta/recipes-support/expand-on-first-boot/expand-on-first-boot_1.2.bb b/meta/recipes-support/expand-on-first-boot/expand-on-first-boot_1.2.bb
index 0996000..fe7b72b 100644
--- a/meta/recipes-support/expand-on-first-boot/expand-on-first-boot_1.2.bb
+++ b/meta/recipes-support/expand-on-first-boot/expand-on-first-boot_1.2.bb
@@ -10,7 +10,7 @@ inherit dpkg-raw
 DESCRIPTION = "This service grows the last partition to the full medium during first boot"
 MAINTAINER = "isar-users <isar-users@googlegroups.com>"
 
-DEBIAN_DEPENDS = "systemd, sed, grep, coreutils, mount, e2fsprogs, fdisk (>=2.29.2-3) | util-linux (<2.29.2-3), util-linux"
+DEBIAN_DEPENDS = "systemd, sed, grep, coreutils, mount, e2fsprogs, fdisk (>=2.29.2-3) | util-linux (<2.29.2-3), util-linux, btrfs-progs"
 
 SRC_URI = " \
     file://expand-on-first-boot.service \
diff --git a/meta/recipes-support/expand-on-first-boot/files/expand-last-partition.sh b/meta/recipes-support/expand-on-first-boot/files/expand-last-partition.sh
index 57055cc..5cd6fc3 100755
--- a/meta/recipes-support/expand-on-first-boot/files/expand-last-partition.sh
+++ b/meta/recipes-support/expand-on-first-boot/files/expand-last-partition.sh
@@ -7,8 +7,25 @@
 #
 # SPDX-License-Identifier: MIT
 
+exitnlog() {
+	ec=$?
+	set +e
+	if [ "$ec" != "0" ]; then
+		echo "ERROR: $0 failed"
+	else
+		echo "$0 succeded"
+	fi >/dev/kmsg
+	if [ "$ec" != "0" -a -e /etc/unsupervised ]; then
+		reboot
+	fi
+}
+export -f exitnlog
+trap exitnlog EXIT
+
 set -e
 
+udevadm settle
+
 ROOT_DEV="$(findmnt / -o source -n)"
 ROOT_DEV_NAME=${ROOT_DEV##*/}
 ROOT_DEV_SLAVE=$(find /sys/block/"${ROOT_DEV_NAME}"/slaves -mindepth 1 -print -quit 2>/dev/null || true)
@@ -16,70 +33,67 @@ if [ -n "${ROOT_DEV_SLAVE}" ]; then
 	ROOT_DEV=/dev/${ROOT_DEV_SLAVE##*/}
 fi
 
-BOOT_DEV="$(echo "${ROOT_DEV}" | sed 's/p\?[0-9]*$//')"
-if [ "${ROOT_DEV}" = "${BOOT_DEV}" ]; then
-	echo "Boot device equals root device - no partitioning found" >&2
-	exit 1
-fi
-
-# this value is in blocks. Normally a block has 512 bytes.
-BUFFER_SIZE=32768
-BOOT_DEV_NAME=${BOOT_DEV##*/}
-DISK_SIZE="$(cat /sys/class/block/"${BOOT_DEV_NAME}"/size)"
-ALL_PARTS_SIZE=0
-for PARTITION in /sys/class/block/"${BOOT_DEV_NAME}"/"${BOOT_DEV_NAME}"*; do
-	PART_SIZE=$(cat "${PARTITION}"/size)
-	ALL_PARTS_SIZE=$((ALL_PARTS_SIZE + PART_SIZE))
-done
-
-MINIMAL_SIZE=$((ALL_PARTS_SIZE + BUFFER_SIZE))
-if [ "$DISK_SIZE" -lt "$MINIMAL_SIZE" ]; then
-	echo "Disk is practically already full, doing nothing." >&2
-	exit 0
-fi
+# full resizing of ext4 and btrfs does not fail nor hurt but supporting more
+# filesystems in future might change this condition, so commenting this code
+useless_for_now() {
+	# this value is in blocks. Normally a block has 512 bytes.
+	BUFFER_SIZE=32768
+	BOOT_DEV_NAME=${BOOT_DEV##*/}
+	DISK_SIZE="$(cat /sys/class/block/"${BOOT_DEV_NAME}"/size)"
+	ALL_PARTS_SIZE=0
+	for PARTITION in /sys/class/block/"${BOOT_DEV_NAME}"/"${BOOT_DEV_NAME}"*; do
+		PART_SIZE=$(cat "${PARTITION}"/size)
+		ALL_PARTS_SIZE=$((ALL_PARTS_SIZE + PART_SIZE))
+	done
 
-LAST_PART="$(sfdisk -d "${BOOT_DEV}" 2>/dev/null | tail -1 | cut -d ' ' -f 1)"
+	MINIMAL_SIZE=$((ALL_PARTS_SIZE + BUFFER_SIZE))
+	if [ "$DISK_SIZE" -lt "$MINIMAL_SIZE" ]; then
+		echo "Disk is practically already full, doing nothing." >&2
+		exit 0
+	fi
+} 
 
-# Transform the partition table as follows:
-#
-# - Remove any 'last-lba' header so sfdisk uses the entire available space.
-# - If this partition table is MBR and an extended partition container (EBR)
-#   exists, we assume this needs to be expanded as well; remove its size
-#   field so sfdisk expands it.
-# - For the previously fetched last partition, also remove the size field so
-#   sfdisk expands it.
-sfdisk -d "${BOOT_DEV}" 2>/dev/null | \
-	grep -v last-lba | \
-	sed 's|^\(.*, \)size=[^,]*, \(type=[f5]\)$|\1\2|' | \
-	sed 's|^\('"${LAST_PART}"' .*, \)size=[^,]*, |\1|' | \
-	sfdisk --force "${BOOT_DEV}"
-
-# Inform the kernel about the partitioning change
-partx -u "${LAST_PART}"
-
-# this is for debian stretch or systemd < 236
-if [ ! -x /lib/systemd/systemd-growfs ]; then
-	# Do not fail resize2fs if no mtab entry is found, e.g.,
-	# when using systemd mount units.
-	export EXT2FS_NO_MTAB_OK=1
+# TODO, 14.12.2022: this change should be tested with a single volume system
+# preliminary tests shown that ext4 and btrfs entire volumes cam be resized 
+BOOT_DEV="$(echo "${ROOT_DEV}" | sed 's/p\?[0-9]*$//')"
+if [ "${ROOT_DEV}" = "${BOOT_DEV}" ]; then
+	LAST_PART="${BOOT_DEV}"
+else
+	LAST_PART="$(sfdisk -d "${BOOT_DEV}" 2>/dev/null | tail -1 | cut -d ' ' -f 1)"
 
-	resize2fs "${LAST_PART}"
-	exit 0
-fi
+	# Transform the partition table as follows:
+	#
+	# - Remove any 'last-lba' header so sfdisk uses the entire available space.
+	# - If this partition table is MBR and an extended partition container (EBR)
+	#   exists, we assume this needs to be expanded as well; remove its size
+	#   field so sfdisk expands it.
+	# - For the previously fetched last partition, also remove the size field so
+	#   sfdisk expands it.
+	sfdisk -d "${BOOT_DEV}" 2>/dev/null | \
+		grep -v last-lba | \
+		sed 's|^\(.*, \)size=[^,]*, \(type=[f5]\)$|\1\2|' | \
+		sed 's|^\('"${LAST_PART}"' .*, \)size=[^,]*, |\1|' | \
+		sfdisk --force "${BOOT_DEV}"
 
-if grep -q x-systemd.growfs /etc/fstab; then
-	echo "Found x-systemd.growfs option in /etc/fstab, won't call it explicitly." >&2
-	exit 0
+	# Inform the kernel about the partitioning change
+	partx -u "${LAST_PART}"
 fi
 
-# mount $LAST_PART out of tree, so we won't conflict with other mounts
-MOUNT_POINT=$(mktemp -d -p /mnt "$(basename "$0").XXXXXXXXXX")
-if [ ! -d "${MOUNT_POINT}" ]; then
-	echo "Cannot create temporary mount point ${MOUNT_POINT}." >&2
-	exit 1
-fi
+# Do not fail resize2fs if no mtab entry is found, e.g.,
+# when using systemd mount units.
+export EXT2FS_NO_MTAB_OK=1
 
-mount "${LAST_PART}" "${MOUNT_POINT}"
-/lib/systemd/systemd-growfs "${MOUNT_POINT}"
-umount "${MOUNT_POINT}"
-rmdir "${MOUNT_POINT}"
+case $(lsblk -fno FSTYPE "${LAST_PART}") in
+	ext[234]) resize2fs "${LAST_PART}"
+		;;
+	btrfs) 	err=0
+		tmpdir=$(mktemp -d -p "$TMPDIR" btrfs.XXXX || \
+			mktemp -d -p "/dev/shm" btrfs.XXXX)
+		mount "${LAST_PART}" "$tmpdir"
+		btrfs filesystem resize max "$tmpdir" || err=1
+		umount "$tmpdir" && rmdir "$tmpdir"
+		exit $err
+		;;
+	*)	echo "WARNING: $0 unsupported filesystem" >/dev/kmsg || true
+		;;
+esac
diff --git a/meta/recipes-support/expand-on-first-boot/files/expand-on-first-boot.service b/meta/recipes-support/expand-on-first-boot/files/expand-on-first-boot.service
index fda5001..2c120c4 100644
--- a/meta/recipes-support/expand-on-first-boot/files/expand-on-first-boot.service
+++ b/meta/recipes-support/expand-on-first-boot/files/expand-on-first-boot.service
@@ -7,7 +7,9 @@
 Description=Expand last partition
 DefaultDependencies=no
 Conflicts=shutdown.target
+Requires=systemd-udevd.service
 After=systemd-remount-fs.service
+After=systemd-udevd.service
 Before=local-fs-pre.target shutdown.target
 ConditionPathIsReadWrite=/etc
 
-- 
2.34.1