From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6818448905053667328 X-Received: by 2002:a17:906:c44b:: with SMTP id ck11mr25607748ejb.110.1587549194120; Wed, 22 Apr 2020 02:53:14 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a17:906:8384:: with SMTP id p4ls706039ejx.3.gmail; Wed, 22 Apr 2020 02:53:13 -0700 (PDT) X-Google-Smtp-Source: APiQypJvq8+2YYr96nbfa2LaQgh5mZZjLKq9mSwnXopurRbfUjfmNDLviMD7FqWDcxEntA2EqfAi X-Received: by 2002:a17:906:3443:: with SMTP id d3mr26225768ejb.18.1587549193536; Wed, 22 Apr 2020 02:53:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1587549193; cv=none; d=google.com; s=arc-20160816; b=EeUSnNygmKfjoY34NR1my+Wad6V0Dktp2z0KQdbFHPClp0SM/U2F/MB8XfxsfXfkkY K3cy0cMW3NMieNecx4b6M5BpI+aBd3DssYNt9rIsIGvyPWCTCmInaCxdlUHu0wRtr1ZG 2uM/D5NOwZRSOhFv3+QD7FJItQqoPdmSbfHb1C22fx1AzlPZ46TiOl99kdxEXmHVmEk0 tlszj+dh/Ocf3DR0+3eeLCstU5QAIApoy4OuXjwN4CYbo9H2fLKyAvUpe4CfpW0nV6W5 nnnbwBQlTrixhhdA79rC7/9I+xTWBwzKx5p7gU7prxG0jRxRh2EBr7aDWsu6bdTaQT5g L/hQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date; bh=VDmYR6RoJMWQ8jKfLKrQT6b+/yNLET+AdTOQMN0hcZE=; b=ij1M9ikJP0ST4v5egABaCxjqvqYMxieNJeamUIPl1v2IoK7f5gPvxEQUbNRMJn12rU PSwWBbeG3LS3i6MLaW8bnUWrbIKCyeclCJBODGw0QJQaxCRGh5+ypOu+lN/eHxbwAdLn ffG+fpTr9CR3WiU4DTfLCEzaLeAiYfIOO9+d6E8yTxSoZHSBh6ifJFZAfaEWV39PXQkC Acw4VjZuA4T3twcs68mS6cbKWO8XJU7Va0W9JapZ92U8UH/zIUXFmuhFw4CCOvUpGXSl 8v1iBF4OZuMhCs3N2d8rxJVqm3e8uHqhAKQ+7j6pklbj+ie+bLLdEhI28JuMZUAZYFH7 1+Yw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from lizzard.sbs.de (lizzard.sbs.de. [194.138.37.39]) by gmr-mx.google.com with ESMTPS id a41si222138edf.1.2020.04.22.02.53.13 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 22 Apr 2020 02:53:13 -0700 (PDT) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.39 as permitted sender) client-ip=194.138.37.39; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by lizzard.sbs.de (8.15.2/8.15.2) with ESMTPS id 03M9rD21030754 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Wed, 22 Apr 2020 11:53:13 +0200 Received: from md1za8fc.ad001.siemens.net ([167.87.23.203]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id 03M9rC6a004266; Wed, 22 Apr 2020 11:53:12 +0200 Date: Wed, 22 Apr 2020 11:53:11 +0200 From: Henning Schild To: "Q. Gylstorff" Cc: , Subject: Re: [PATCH] meta/recipes-support: Mount overlay-fs for /etc Message-ID: <20200422115311.750c2ff1@md1za8fc.ad001.siemens.net> In-Reply-To: <20200422082403.7392-1-Quirin.Gylstorff@siemens.com> References: <20200422082403.7392-1-Quirin.Gylstorff@siemens.com> X-Mailer: Claws Mail 3.17.5 (GTK+ 2.24.32; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-TUID: pFBEQhtfuhbE Identifying changing content that needs to be persisted is a tough task, and it goes much further than etc. In fact etc is the least relevant for an Isar image. Because it usually does not really change at runtime, maybe except for a first on-first-boot services that disable themselfs. What changes outside of isar ... in the worst case because users postprocess or manually opererate ... can not be covered by isar. Debian already protects etc if you choose to update with packages. If you choose to update a full rootfs and want to persist important stuff ... /var and /home are way more interesting than /etc. That would need to be covered as well ... I think without a full concept for a read-only rootfs its pointless to start with the low-hanging etc. Henning On Wed, 22 Apr 2020 10:24:03 +0200 "Q. Gylstorff" wrote: > From: Quirin Gylstorff > > Add an overlay fs[1] mount for /etc. This overlay allows an image > update of root file-system without overwriting the user settings in > /etc. The overlay mounts during the systemd target > `local-fs-pre.target.` Due to this target the by systemd generated > fsck service is not a dependency for the overlay. > > [1]: > https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt > > Signed-off-by: Quirin Gylstorff > --- > doc/user_manual.md | 26 ++++++ > meta-isar/conf/machine/qemuamd64.conf | 4 +- > .../etc-overlay/etc-overlay_0.1.bb | 39 +++++++++ > .../etc-overlay/files/etc-hostname.service | 14 ++++ > .../etc-overlay/files/etc.mount | 15 ++++ > .../files/overlay-parse-etc.service | 14 ++++ > .../etc-overlay/files/ovl.mount.tmpl | 14 ++++ > .../etc-overlay/files/postinst | 32 +++++++ > meta/recipes-support/etc-overlay/files/postrm | 29 +++++++ > scripts/lib/wic/canned-wks/etc-overlay.inc | 5 ++ > scripts/lib/wic/plugins/source/etc-overlay.py | 84 > +++++++++++++++++++ 11 files changed, 275 insertions(+), 1 deletion(-) > create mode 100644 > meta/recipes-support/etc-overlay/etc-overlay_0.1.bb create mode > 100644 meta/recipes-support/etc-overlay/files/etc-hostname.service > create mode 100644 meta/recipes-support/etc-overlay/files/etc.mount > create mode 100644 > meta/recipes-support/etc-overlay/files/overlay-parse-etc.service > create mode 100644 > meta/recipes-support/etc-overlay/files/ovl.mount.tmpl create mode > 100755 meta/recipes-support/etc-overlay/files/postinst create mode > 100644 meta/recipes-support/etc-overlay/files/postrm create mode > 100644 scripts/lib/wic/canned-wks/etc-overlay.inc create mode 100644 > scripts/lib/wic/plugins/source/etc-overlay.py > > diff --git a/doc/user_manual.md b/doc/user_manual.md > index d13a74e..5ec7003 100644 > --- a/doc/user_manual.md > +++ b/doc/user_manual.md > @@ -901,3 +901,29 @@ And build the corresponding image target: > ``` > bitbake mc:qemuarm64-buster:isar-image-base > ``` > +## Mount overlay for /etc > + > +### Motivation > + > +If a software update solution updates the complete image, the > content of `/etc` +is also set to new image content. In this case, > the user needs to reconfigure +her system. > + > +### Approach/Solution > + > +A possible solution is to create an additional partition, which > superimposes +`/etc` with an [overlay > file-system](https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt). > + +### Example > + > +Add the following lines to local.conf: > +``` > +IMAGE_TYPE = "wic-img" > +WKS_FILE = "sdimage-efi-overlay" > +IMAGE_INSTALL += "etc-overlay" > +``` > +And build the corresponding image target: > + > +``` > +bitbake mc:qemuamd64-buster:isar-image-base > +``` > diff --git a/meta-isar/conf/machine/qemuamd64.conf > b/meta-isar/conf/machine/qemuamd64.conf index 028b738..d429d96 100644 > --- a/meta-isar/conf/machine/qemuamd64.conf > +++ b/meta-isar/conf/machine/qemuamd64.conf > @@ -6,11 +6,13 @@ DISTRO_ARCH ?= "amd64" > KERNEL_NAME ?= "amd64" > > IMAGE_TYPE ?= "wic-img" > -WKS_FILE ?= "sdimage-efi" > +WKS_FILE ?= "sdimage-efi-overlay" > IMAGER_INSTALL += "${GRUB_BOOTLOADER_INSTALL}" > > IMAGE_INSTALL += "sshd-regen-keys" > > +IMAGE_INSTALL += "etc-overlay" > + > QEMU_ARCH ?= "x86_64" > QEMU_MACHINE ?= "q35" > QEMU_CPU ?= "" > diff --git a/meta/recipes-support/etc-overlay/etc-overlay_0.1.bb > b/meta/recipes-support/etc-overlay/etc-overlay_0.1.bb new file mode > 100644 index 0000000..cbed71d > --- /dev/null > +++ b/meta/recipes-support/etc-overlay/etc-overlay_0.1.bb > @@ -0,0 +1,39 @@ > +# Create a overlay for /etc to freeze a default configuration > +# > +# This software is a part of ISAR. > +# Copyright (c) Siemens AG, 2020 > +# > +# SPDX-License-Identifier: MIT > + > + > +DESCRIPTION = "overlay systemd-mount" > + > +DEBIAN_DEPENDS = "systemd" > + > +SRC_URI = "file://postinst \ > + file://postrm \ > + file://etc.mount \ > + file://ovl.mount.tmpl \ > + file://overlay-parse-etc.service \ > + file://etc-hostname.service" > + > +FS_COMMIT_INTERVAL ?= "20" > + > +TEMPLATE_VARS += "FS_COMMIT_INTERVAL" > +TEMPLATE_FILES += "ovl.mount.tmpl" > + > +inherit dpkg-raw > + > +do_install() { > + install -m 0755 -d ${D}/ovl > + touch ${D}/ovl/.keep > + > + TARGET=${D}/lib/systemd/system > + install -m 0755 -d ${TARGET} > + install -m 0644 ${WORKDIR}/etc.mount ${TARGET}/etc.mount > + install -m 0644 ${WORKDIR}/ovl.mount ${TARGET}/ovl.mount > + install -m 0644 ${WORKDIR}/overlay-parse-etc.service > ${TARGET}/overlay-parse-etc.service > + install -m 0644 ${WORKDIR}/etc-hostname.service > ${TARGET}/etc-hostname.service +} > + > +addtask do_install after do_transform_template > diff --git > a/meta/recipes-support/etc-overlay/files/etc-hostname.service > b/meta/recipes-support/etc-overlay/files/etc-hostname.service new > file mode 100644 index 0000000..2306b9f --- /dev/null > +++ b/meta/recipes-support/etc-overlay/files/etc-hostname.service > @@ -0,0 +1,14 @@ > +[Unit] > +Description=set hostname /etc overlay-aware > +Before=network-pre.target > +Wants=network-pre.target > +Requires=etc.mount > +After=etc.mount > + > +[Service] > +Type=oneshot > +RemainAfterExit=yes > +ExecStart=/bin/hostname --boot --file /etc/hostname > + > +[Install] > +WantedBy=basic.target > diff --git a/meta/recipes-support/etc-overlay/files/etc.mount > b/meta/recipes-support/etc-overlay/files/etc.mount new file mode > 100644 index 0000000..59ee0d7 > --- /dev/null > +++ b/meta/recipes-support/etc-overlay/files/etc.mount > @@ -0,0 +1,15 @@ > +[Unit] > +Description=Overlay-mount /etc > +DefaultDependencies=no > +Before=local-fs-pre.target > +Requires=ovl.mount > +After=ovl.mount > + > +[Mount] > +What=overlay > +Where=/etc > +Type=overlay > +Options=noauto,x-systemd.automount,lowerdir=/etc,upperdir=/ovl/etc,workdir=/ovl/.atomic > + > +[Install] > +WantedBy=local-fs-pre.target > diff --git > a/meta/recipes-support/etc-overlay/files/overlay-parse-etc.service > b/meta/recipes-support/etc-overlay/files/overlay-parse-etc.service > new file mode 100644 index 0000000..eb5d422 --- /dev/null > +++ b/meta/recipes-support/etc-overlay/files/overlay-parse-etc.service > @@ -0,0 +1,14 @@ > +[Unit] > +Description=Reload Configuration from the etc overlay > +DefaultDependencies=no > +Requires=etc.mount > +After=etc.mount > +Before=local-fs-pre.target > + > +[Service] > +Type=oneshot > +RemainAfterExit=yes > +ExecStartPre=!/bin/systemctl daemon-reload > +ExecStart=!/bin/systemctl --no-block isolate multi-user.target > +[Install] > +WantedBy=local-fs-pre.target > diff --git a/meta/recipes-support/etc-overlay/files/ovl.mount.tmpl > b/meta/recipes-support/etc-overlay/files/ovl.mount.tmpl new file mode > 100644 index 0000000..3593c55 > --- /dev/null > +++ b/meta/recipes-support/etc-overlay/files/ovl.mount.tmpl > @@ -0,0 +1,14 @@ > +[Unit] > +Description=Mount /etc overlay backing store > +DefaultDependencies=no > +Before=local-fs-pre.target > +Before=etc.mount > + > +[Mount] > +What=/dev/disk/by-partlabel/etcovl > +Where=/ovl > +Type=ext4 > +Options=noatime,nodiratime,data=journal,commit=${FS_COMMIT_INTERVAL},nodelalloc > + > +[Install] > +WantedBy=local-fs-pre.target > diff --git a/meta/recipes-support/etc-overlay/files/postinst > b/meta/recipes-support/etc-overlay/files/postinst new file mode 100755 > index 0000000..7a10d68 > --- /dev/null > +++ b/meta/recipes-support/etc-overlay/files/postinst > @@ -0,0 +1,32 @@ > +#!/bin/sh > +# postinst script for etc-overlay > +# > +# see: dh_installdeb(1) > + > +set -e > + > +case "$1" in > + configure) > + deb-systemd-helper enable etc.mount || true > + deb-systemd-helper enable ovl.mount || true > + deb-systemd-helper enable overlay-parse-etc.service || true > + deb-systemd-helper enable systemd-remount-fs.service || true > + deb-systemd-helper enable etc-hostname.service || true > + ;; > + > + abort-upgrade|abort-remove|abort-deconfigure) > + ;; > + > + *) > + echo "postinst called with unknown argument \`$1'" >&2 > + exit 1 > + ;; > +esac > + > +# dh_installdeb will replace this with shell code automatically > +# generated by other debhelper scripts. > + > +#DEBHELPER# > + > +exit 0 > + > diff --git a/meta/recipes-support/etc-overlay/files/postrm > b/meta/recipes-support/etc-overlay/files/postrm new file mode 100644 > index 0000000..7a3defb > --- /dev/null > +++ b/meta/recipes-support/etc-overlay/files/postrm > @@ -0,0 +1,29 @@ > +#!/bin/sh > +# postrm script for etc-overlay > +# > +# see: dh_installdeb(1) > + > +set -e > + > + > +case "$1" in > + > purge|remove|upgrade|failed-upgrade|abort-install|abort-upgrade|disappear) > + deb-systemd-helper disable etc.mount || true > + deb-systemd-helper disable ovl.mount || true > + deb-systemd-helper disable overlay-parse-etc.service || true > + deb-systemd-helper disable systemd-remount-fs.service || true > + deb-systemd-helper disable etc-hostname.service || true > + ;; > + > + *) > + echo "postrm called with unknown argument \`$1'" >&2 > + exit 1 > + ;; > +esac > + > +# dh_installdeb will replace this with shell code automatically > +# generated by other debhelper scripts. > + > +#DEBHELPER# > + > +exit 0 > diff --git a/scripts/lib/wic/canned-wks/etc-overlay.inc > b/scripts/lib/wic/canned-wks/etc-overlay.inc new file mode 100644 > index 0000000..7b40854 > --- /dev/null > +++ b/scripts/lib/wic/canned-wks/etc-overlay.inc > @@ -0,0 +1,5 @@ > +# add a overlay partition to the image > + > +# overlay partition > +part --source etc-overlay --ondisk sda --size 100M --extra-space > 128M --overhead-factor 1 --label etcovl --align 1024 --fstype=ext4 + > diff --git a/scripts/lib/wic/plugins/source/etc-overlay.py > b/scripts/lib/wic/plugins/source/etc-overlay.py new file mode 100644 > index 0000000..55189a2 > --- /dev/null > +++ b/scripts/lib/wic/plugins/source/etc-overlay.py > @@ -0,0 +1,84 @@ > +# ex:ts=4:sw=4:sts=4:et > +# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > +# > +# Copyright (c) 2014, Intel Corporation. > +# Copyright (c) 2018, Siemens AG. > +# All rights reserved. > +# > +# This program is free software; you can redistribute it and/or > modify +# it under the terms of the GNU General Public License > version 2 as +# published by the Free Software Foundation. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License > along +# with this program; if not, write to the Free Software > Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. +# > +# DESCRIPTION > +# This implements the 'etc-overlay' source plugin class for 'wic' > +# > +# AUTHORS > +# Tom Zanussi > +# Andreas Reichel > +# Quirin Gylstorff > + > + > +import logging > + > +msger = logging.getLogger('wic') > + > +from wic.pluginbase import SourcePlugin > +from wic.utils.misc import exec_cmd,BOOTDD_EXTRA_SPACE > + > +class EtcOverlayPlugin(SourcePlugin): > + """ > + Create an overlay file system scheme for etc > + """ > + > + name = 'etc-overlay' > + > + @classmethod > + def do_prepare_partition(cls, part, source_params, creator, > cr_workdir, > + oe_builddir, deploy_dir, kernel_dir, > + rootfs_dir, native_sysroot): > + > + part_rootfs_dir = "%s/disk/%s.%s" % (cr_workdir, > + part.label, > + part.lineno) > + create_dir_cmd = "install -d %s" % part_rootfs_dir > + exec_cmd(create_dir_cmd) > + > + exec_cmd("install -m 0755 -d %s/etc" % part_rootfs_dir) > + exec_cmd("install -m 0755 -d %s/.atomic" % part_rootfs_dir) > + > + blocks = 16 > + extra_blocks = part.get_extra_block_count(blocks) > + if extra_blocks < BOOTDD_EXTRA_SPACE: > + extra_blocks = BOOTDD_EXTRA_SPACE > + blocks += extra_blocks > + blocks = blocks + (16 - (blocks % 16)) > + > + msger.debug("Added %d extra blocks to %s to get to %d total > blocks", > + extra_blocks, part.mountpoint, blocks) > + > + # ext4 image, created by mkfs.ext4 > + etcovlimg = "%s/%s.%s.img" % (cr_workdir, part.label, > part.lineno) > + partfs_cmd = "dd if=/dev/zero of=%s bs=512 count=%d" % > (etcovlimg, > + > blocks) > + exec_cmd(partfs_cmd) > + > + partfs_cmd = "mkfs.ext4 %s -d %s" % (etcovlimg, > part_rootfs_dir) > + exec_cmd(partfs_cmd) > + > + chmod_cmd = "chmod 644 %s" % etcovlimg > + exec_cmd(chmod_cmd) > + > + du_cmd = "du -Lbks %s" % etcovlimg > + etcovlimg_size = int(exec_cmd(du_cmd).split()[0]) > + > + part.size = etcovlimg_size > + part.source_file = etcovlimg