Re: [PATCH] meta/recipes-support: Mount overlay-fs for /etc

[Jan Kiszka <jan.kiszka@siemens.com>]

On 22.04.20 13:00, Gylstorff Quirin wrote:
> 
> 
> On 4/22/20 10:30 AM, [ext] Jan Kiszka wrote:
>> On 22.04.20 10:24, Q. Gylstorff wrote:
>>> From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>>
>>> Add an overlay fs[1] mount for /etc. This overlay allows an image 
>>> update of
>>> root file-system without overwriting the user settings in /etc. The 
>>> overlay
>>> mounts during the systemd target `local-fs-pre.target.` Due to this 
>>> target
>>> the by systemd generated fsck service is not a dependency for the 
>>> overlay.
>>>
>>> [1]: https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt
>>>
>>> Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>> ---
>>>   doc/user_manual.md                            | 26 ++++++
>>>   meta-isar/conf/machine/qemuamd64.conf         |  4 +-
>>>   .../etc-overlay/etc-overlay_0.1.bb            | 39 +++++++++
>>>   .../etc-overlay/files/etc-hostname.service    | 14 ++++
>>>   .../etc-overlay/files/etc.mount               | 15 ++++
>>>   .../files/overlay-parse-etc.service           | 14 ++++
>>>   .../etc-overlay/files/ovl.mount.tmpl          | 14 ++++
>>>   .../etc-overlay/files/postinst                | 32 +++++++
>>>   meta/recipes-support/etc-overlay/files/postrm | 29 +++++++
>>>   scripts/lib/wic/canned-wks/etc-overlay.inc    |  5 ++
>>>   scripts/lib/wic/plugins/source/etc-overlay.py | 84 +++++++++++++++++++
>>>   11 files changed, 275 insertions(+), 1 deletion(-)
>>>   create mode 100644 meta/recipes-support/etc-overlay/etc-overlay_0.1.bb
>>>   create mode 100644 
>>> meta/recipes-support/etc-overlay/files/etc-hostname.service
>>>   create mode 100644 meta/recipes-support/etc-overlay/files/etc.mount
>>>   create mode 100644 
>>> meta/recipes-support/etc-overlay/files/overlay-parse-etc.service
>>>   create mode 100644 
>>> meta/recipes-support/etc-overlay/files/ovl.mount.tmpl
>>>   create mode 100755 meta/recipes-support/etc-overlay/files/postinst
>>>   create mode 100644 meta/recipes-support/etc-overlay/files/postrm
>>>   create mode 100644 scripts/lib/wic/canned-wks/etc-overlay.inc
>>>   create mode 100644 scripts/lib/wic/plugins/source/etc-overlay.py
>>>
>>> diff --git a/doc/user_manual.md b/doc/user_manual.md
>>> index d13a74e..5ec7003 100644
>>> --- a/doc/user_manual.md
>>> +++ b/doc/user_manual.md
>>> @@ -901,3 +901,29 @@ And build the corresponding image target:
>>>   ```
>>>   bitbake mc:qemuarm64-buster:isar-image-base
>>>   ```
>>> +## Mount overlay for /etc
>>> +
>>> +### Motivation
>>> +
>>> +If a software update solution updates the complete image, the 
>>> content of `/etc`
>>> +is also set to new image content. In this case, the user needs to 
>>> reconfigure
>>> +her system.
>>> +
>>> +### Approach/Solution
>>> +
>>> +A possible solution is to create an additional partition, which 
>>> superimposes
>>> +`/etc` with an [overlay 
>>> file-system](https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt). 
>>>
>>> +
>>> +### Example
>>> +
>>> +Add the following lines to local.conf:
>>> +```
>>> +IMAGE_TYPE = "wic-img"
>>> +WKS_FILE = "sdimage-efi-overlay"
>>> +IMAGE_INSTALL += "etc-overlay"
>>> +```
>>> +And build the corresponding image target:
>>> +
>>> +```
>>> +bitbake mc:qemuamd64-buster:isar-image-base
>>> +```
>>> diff --git a/meta-isar/conf/machine/qemuamd64.conf 
>>> b/meta-isar/conf/machine/qemuamd64.conf
>>> index 028b738..d429d96 100644
>>> --- a/meta-isar/conf/machine/qemuamd64.conf
>>> +++ b/meta-isar/conf/machine/qemuamd64.conf
>>> @@ -6,11 +6,13 @@ DISTRO_ARCH ?= "amd64"
>>>   KERNEL_NAME ?= "amd64"
>>>   IMAGE_TYPE ?= "wic-img"
>>> -WKS_FILE ?= "sdimage-efi"
>>> +WKS_FILE ?= "sdimage-efi-overlay"
>>>   IMAGER_INSTALL += "${GRUB_BOOTLOADER_INSTALL}"
>>>   IMAGE_INSTALL += "sshd-regen-keys"
>>> +IMAGE_INSTALL += "etc-overlay"
>>> +
>>>   QEMU_ARCH ?= "x86_64"
>>>   QEMU_MACHINE ?= "q35"
>>>   QEMU_CPU ?= ""
>>> diff --git a/meta/recipes-support/etc-overlay/etc-overlay_0.1.bb 
>>> b/meta/recipes-support/etc-overlay/etc-overlay_0.1.bb
>>> new file mode 100644
>>> index 0000000..cbed71d
>>> --- /dev/null
>>> +++ b/meta/recipes-support/etc-overlay/etc-overlay_0.1.bb
>>> @@ -0,0 +1,39 @@
>>> +# Create a overlay for /etc to freeze a default configuration
>>> +#
>>> +# This software is a part of ISAR.
>>> +# Copyright (c) Siemens AG, 2020
>>> +#
>>> +# SPDX-License-Identifier: MIT
>>> +
>>> +
>>> +DESCRIPTION = "overlay systemd-mount"
>>> +
>>> +DEBIAN_DEPENDS = "systemd"
>>> +
>>> +SRC_URI = "file://postinst \
>>> +           file://postrm \
>>> +           file://etc.mount \
>>> +           file://ovl.mount.tmpl \
>>> +           file://overlay-parse-etc.service \
>>> +           file://etc-hostname.service"
>>> +
>>> +FS_COMMIT_INTERVAL ?= "20"
>>> +
>>> +TEMPLATE_VARS  += "FS_COMMIT_INTERVAL"
>>> +TEMPLATE_FILES += "ovl.mount.tmpl"
>>> +
>>> +inherit dpkg-raw
>>> +
>>> +do_install() {
>>> +    install -m 0755 -d ${D}/ovl
>>> +    touch ${D}/ovl/.keep
>>> +
>>> +    TARGET=${D}/lib/systemd/system
>>> +    install -m 0755 -d ${TARGET}
>>> +    install -m 0644 ${WORKDIR}/etc.mount ${TARGET}/etc.mount
>>> +    install -m 0644 ${WORKDIR}/ovl.mount ${TARGET}/ovl.mount
>>> +    install -m 0644 ${WORKDIR}/overlay-parse-etc.service 
>>> ${TARGET}/overlay-parse-etc.service
>>> +    install -m 0644 ${WORKDIR}/etc-hostname.service 
>>> ${TARGET}/etc-hostname.service
>>> +}
>>> +
>>> +addtask do_install after do_transform_template
>>> diff --git 
>>> a/meta/recipes-support/etc-overlay/files/etc-hostname.service 
>>> b/meta/recipes-support/etc-overlay/files/etc-hostname.service
>>> new file mode 100644
>>> index 0000000..2306b9f
>>> --- /dev/null
>>> +++ b/meta/recipes-support/etc-overlay/files/etc-hostname.service
>>> @@ -0,0 +1,14 @@
>>> +[Unit]
>>> +Description=set hostname /etc overlay-aware
>>> +Before=network-pre.target
>>> +Wants=network-pre.target
>>> +Requires=etc.mount
>>> +After=etc.mount
>>> +
>>> +[Service]
>>> +Type=oneshot
>>> +RemainAfterExit=yes
>>> +ExecStart=/bin/hostname --boot --file /etc/hostname
>>> +
>>> +[Install]
>>> +WantedBy=basic.target
>>> diff --git a/meta/recipes-support/etc-overlay/files/etc.mount 
>>> b/meta/recipes-support/etc-overlay/files/etc.mount
>>> new file mode 100644
>>> index 0000000..59ee0d7
>>> --- /dev/null
>>> +++ b/meta/recipes-support/etc-overlay/files/etc.mount
>>> @@ -0,0 +1,15 @@
>>> +[Unit]
>>> +Description=Overlay-mount /etc
>>> +DefaultDependencies=no
>>> +Before=local-fs-pre.target
>>> +Requires=ovl.mount
>>> +After=ovl.mount
>>> +
>>> +[Mount]
>>> +What=overlay
>>> +Where=/etc
>>> +Type=overlay
>>> +Options=noauto,x-systemd.automount,lowerdir=/etc,upperdir=/ovl/etc,workdir=/ovl/.atomic 
>>>
>>> +
>>> +[Install]
>>> +WantedBy=local-fs-pre.target
>>> diff --git 
>>> a/meta/recipes-support/etc-overlay/files/overlay-parse-etc.service 
>>> b/meta/recipes-support/etc-overlay/files/overlay-parse-etc.service
>>> new file mode 100644
>>> index 0000000..eb5d422
>>> --- /dev/null
>>> +++ b/meta/recipes-support/etc-overlay/files/overlay-parse-etc.service
>>> @@ -0,0 +1,14 @@
>>> +[Unit]
>>> +Description=Reload Configuration from the etc overlay
>>> +DefaultDependencies=no
>>> +Requires=etc.mount
>>> +After=etc.mount
>>> +Before=local-fs-pre.target
>>> +
>>> +[Service]
>>> +Type=oneshot
>>> +RemainAfterExit=yes
>>> +ExecStartPre=!/bin/systemctl daemon-reload
>>> +ExecStart=!/bin/systemctl --no-block isolate multi-user.target
>>> +[Install]
>>> +WantedBy=local-fs-pre.target
>>> diff --git a/meta/recipes-support/etc-overlay/files/ovl.mount.tmpl 
>>> b/meta/recipes-support/etc-overlay/files/ovl.mount.tmpl
>>> new file mode 100644
>>> index 0000000..3593c55
>>> --- /dev/null
>>> +++ b/meta/recipes-support/etc-overlay/files/ovl.mount.tmpl
>>> @@ -0,0 +1,14 @@
>>> +[Unit]
>>> +Description=Mount /etc overlay backing store
>>> +DefaultDependencies=no
>>> +Before=local-fs-pre.target
>>> +Before=etc.mount
>>> +
>>> +[Mount]
>>> +What=/dev/disk/by-partlabel/etcovl
>>> +Where=/ovl
>>> +Type=ext4
>>> +Options=noatime,nodiratime,data=journal,commit=${FS_COMMIT_INTERVAL},nodelalloc 
>>>
>>> +
>>> +[Install]
>>> +WantedBy=local-fs-pre.target
>>> diff --git a/meta/recipes-support/etc-overlay/files/postinst 
>>> b/meta/recipes-support/etc-overlay/files/postinst
>>> new file mode 100755
>>> index 0000000..7a10d68
>>> --- /dev/null
>>> +++ b/meta/recipes-support/etc-overlay/files/postinst
>>> @@ -0,0 +1,32 @@
>>> +#!/bin/sh
>>> +# postinst script for etc-overlay
>>> +#
>>> +# see: dh_installdeb(1)
>>> +
>>> +set -e
>>> +
>>> +case "$1" in
>>> +    configure)
>>> +        deb-systemd-helper enable etc.mount  || true
>>> +        deb-systemd-helper enable ovl.mount  || true
>>> +        deb-systemd-helper enable overlay-parse-etc.service || true
>>> +        deb-systemd-helper enable systemd-remount-fs.service || true
>>> +        deb-systemd-helper enable etc-hostname.service || true
>>> +    ;;
>>> +
>>> +    abort-upgrade|abort-remove|abort-deconfigure)
>>> +    ;;
>>> +
>>> +    *)
>>> +        echo "postinst called with unknown argument \`$1'" >&2
>>> +        exit 1
>>> +        ;;
>>> +esac
>>> +
>>> +# dh_installdeb will replace this with shell code automatically
>>> +# generated by other debhelper scripts.
>>> +
>>> +#DEBHELPER#
>>> +
>>> +exit 0
>>> +
>>> diff --git a/meta/recipes-support/etc-overlay/files/postrm 
>>> b/meta/recipes-support/etc-overlay/files/postrm
>>> new file mode 100644
>>> index 0000000..7a3defb
>>> --- /dev/null
>>> +++ b/meta/recipes-support/etc-overlay/files/postrm
>>> @@ -0,0 +1,29 @@
>>> +#!/bin/sh
>>> +# postrm script for etc-overlay
>>> +#
>>> +# see: dh_installdeb(1)
>>> +
>>> +set -e
>>> +
>>> +
>>> +case "$1" in
>>> + 
>>> purge|remove|upgrade|failed-upgrade|abort-install|abort-upgrade|disappear) 
>>>
>>> +        deb-systemd-helper disable etc.mount  || true
>>> +        deb-systemd-helper disable ovl.mount  || true
>>> +        deb-systemd-helper disable overlay-parse-etc.service || true
>>> +        deb-systemd-helper disable systemd-remount-fs.service || true
>>> +        deb-systemd-helper disable etc-hostname.service || true
>>> +     ;;
>>> +
>>> +    *)
>>> +        echo "postrm called with unknown argument \`$1'" >&2
>>> +        exit 1
>>> +    ;;
>>> +esac
>>> +
>>> +# dh_installdeb will replace this with shell code automatically
>>> +# generated by other debhelper scripts.
>>> +
>>> +#DEBHELPER#
>>> +
>>> +exit 0
>>> diff --git a/scripts/lib/wic/canned-wks/etc-overlay.inc 
>>> b/scripts/lib/wic/canned-wks/etc-overlay.inc
>>> new file mode 100644
>>> index 0000000..7b40854
>>> --- /dev/null
>>> +++ b/scripts/lib/wic/canned-wks/etc-overlay.inc
>>> @@ -0,0 +1,5 @@
>>> +# add a overlay partition to the image
>>> +
>>> +# overlay partition
>>> +part --source etc-overlay --ondisk sda --size 100M --extra-space 
>>> 128M --overhead-factor 1 --label etcovl --align 1024 --fstype=ext4
>>> +
>>
>> Where is this used/tested? I think you should create a test/demo 
>> target that uses a read-only rootfs and such an overlay partition via 
>> the include.
>>
>> Jan
> 
> In the patch series it is added to the qemuamd64 target as test. It 
> replaces the sdimage-efi with sdimage-efi-overlay. But a read only 
> rootfs/target is better.
> 

You forgot to include sdimage-efi-overlay.wks - that's why I got 
confused. Or you didn't send all patches?

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux