From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6818448905053667328 X-Received: by 2002:a1c:35c3:: with SMTP id c186mr10410291wma.66.1587553568508; Wed, 22 Apr 2020 04:06:08 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a1c:4088:: with SMTP id n130ls1845685wma.2.canary-gmail; Wed, 22 Apr 2020 04:06:07 -0700 (PDT) X-Google-Smtp-Source: APiQypL0g08/2NBn+SFEoxHtC/8GSoNr1W9Kp40Iu2CiU181RQaca1HsQwvPOedHlPenE0T8zOuJ X-Received: by 2002:a1c:8049:: with SMTP id b70mr10069459wmd.162.1587553567846; Wed, 22 Apr 2020 04:06:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1587553567; cv=none; d=google.com; s=arc-20160816; b=yEMxo9/vGknToryg3huqiPcy/QmDiMmri72lJnmCrCkK9IC0y3ydelITvGsbLNs7vU hFB/kCt04+fVWsNK/JlyOYtd8GnqQlMvqqkt5Kw9ob2BmgHeP2I2AvaPHshcuaCRMpIk XwuhbrxKRjMHRMWpEZv3isjFnr2qs6xtf0VVbnKAkdTEBw7CDt6yS7XxrtOZ6bRxM2xQ ukwYn6osKryM2y/rP/DQrePAQPRL0Vzh0oyb6JH4txLGbywHpJJNyySMBaOM9ffody1o RNpR64rhn/JovpZNZL9SYvt6p8hFBUGaLiTvJnb9DDwFHGAMt2RKBlrC0vWd3lpbdVad 09xQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date; bh=3pCWCpsoZmwCPO3XF8l7AELlsAX5d1j9KjjxriMbczc=; b=LOGKFWFPwmDl68kS+TyL3tWgQpJs1jw50YxUDirs0TpqpOn2XNJpZ+2NEL5TaVdQIM /blCdJczv7ldY01BWoVPurckxZe6sC6x1mqSL5gMaolUtV/+dVVpkW8UYjij9Unw6nvE 6RO2B36131tjV8noJ8wV/qm2Z6mi8SzZy+CvDG4rz8LhVwfVtLeeFTIw2rxmEJnjLBGc 4ypCSDhOtmjgiZNcW/X2bHqEzsKzTYuKqsUSXSVelXVWHqluekqQLy+GbOY+682w4pLb v5zGL9pTLVJQE0Xa9X8QRlYaoXdzzk7zXQuRpR+/gQOtWI9uxMzWvUfMYbQrrCsc2ySk 874w== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from lizzard.sbs.de (lizzard.sbs.de. [194.138.37.39]) by gmr-mx.google.com with ESMTPS id a5si361369wrg.4.2020.04.22.04.06.07 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 22 Apr 2020 04:06:07 -0700 (PDT) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.39 as permitted sender) client-ip=194.138.37.39; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by lizzard.sbs.de (8.15.2/8.15.2) with ESMTPS id 03MB67Ns026583 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Wed, 22 Apr 2020 13:06:07 +0200 Received: from md1za8fc.ad001.siemens.net ([167.87.23.203]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id 03MB6608001966; Wed, 22 Apr 2020 13:06:07 +0200 Date: Wed, 22 Apr 2020 13:06:06 +0200 From: Henning Schild To: Gylstorff Quirin Cc: , Subject: Re: [PATCH] meta/recipes-support: Mount overlay-fs for /etc Message-ID: <20200422130606.0ee639fd@md1za8fc.ad001.siemens.net> In-Reply-To: <8538b97d-792d-47bd-6f92-3ecea68a4869@siemens.com> References: <20200422082403.7392-1-Quirin.Gylstorff@siemens.com> <20200422115311.750c2ff1@md1za8fc.ad001.siemens.net> <8538b97d-792d-47bd-6f92-3ecea68a4869@siemens.com> X-Mailer: Claws Mail 3.17.5 (GTK+ 2.24.32; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-TUID: tVpalbgN540T On Wed, 22 Apr 2020 12:56:28 +0200 Gylstorff Quirin wrote: > On 4/22/20 11:53 AM, Henning Schild wrote: > > Identifying changing content that needs to be persisted is a tough > > task, and it goes much further than etc. In fact etc is the least > > relevant for an Isar image. Because it usually does not really > > change at runtime, maybe except for a first on-first-boot services > > that disable themselfs. > > What changes outside of isar ... in the worst case because users > > postprocess or manually opererate ... can not be covered by isar. > > > > Debian already protects etc if you choose to update with packages. > > If you choose to update a full rootfs and want to persist important > > stuff ... /var and /home are way more interesting than /etc. That > > would need to be covered as well ... > > > > I think without a full concept for a read-only rootfs its pointless > > to start with the low-hanging etc. > > > > Henning > > This feature can be used for read-only fs, but image replacements > updates as written in the doc and commit message. > And the customer use case was etc but I can easily add /home and /var > to the overlay as it is more from the same. What to persist is highly dependant on the image you build and the components to install and run. /etc might be a good first step and showing it upstream early is good. But is it far from complete and quickly jumping on /var or /tmp and declare it "solved" will probably not work. It is not all of var and not all of home, but very specific sub-folders in there. My suggestion would be to first learn more about how to make a debian get a read-only rootfs and what to overlay for "full" persistance. From that we can derive a strategy and later an implementation. Henning > Quirin > > > > > > On Wed, 22 Apr 2020 10:24:03 +0200 > > "Q. Gylstorff" wrote: > > > >> From: Quirin Gylstorff > >> > >> Add an overlay fs[1] mount for /etc. This overlay allows an image > >> update of root file-system without overwriting the user settings in > >> /etc. The overlay mounts during the systemd target > >> `local-fs-pre.target.` Due to this target the by systemd generated > >> fsck service is not a dependency for the overlay. > >> > >> [1]: > >> https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt > >> > >> Signed-off-by: Quirin Gylstorff > >> --- > >> doc/user_manual.md | 26 ++++++ > >> meta-isar/conf/machine/qemuamd64.conf | 4 +- > >> .../etc-overlay/etc-overlay_0.1.bb | 39 +++++++++ > >> .../etc-overlay/files/etc-hostname.service | 14 ++++ > >> .../etc-overlay/files/etc.mount | 15 ++++ > >> .../files/overlay-parse-etc.service | 14 ++++ > >> .../etc-overlay/files/ovl.mount.tmpl | 14 ++++ > >> .../etc-overlay/files/postinst | 32 +++++++ > >> meta/recipes-support/etc-overlay/files/postrm | 29 +++++++ > >> scripts/lib/wic/canned-wks/etc-overlay.inc | 5 ++ > >> scripts/lib/wic/plugins/source/etc-overlay.py | 84 > >> +++++++++++++++++++ 11 files changed, 275 insertions(+), 1 > >> deletion(-) create mode 100644 > >> meta/recipes-support/etc-overlay/etc-overlay_0.1.bb create mode > >> 100644 meta/recipes-support/etc-overlay/files/etc-hostname.service > >> create mode 100644 meta/recipes-support/etc-overlay/files/etc.mount > >> create mode 100644 > >> meta/recipes-support/etc-overlay/files/overlay-parse-etc.service > >> create mode 100644 > >> meta/recipes-support/etc-overlay/files/ovl.mount.tmpl create mode > >> 100755 meta/recipes-support/etc-overlay/files/postinst create mode > >> 100644 meta/recipes-support/etc-overlay/files/postrm create mode > >> 100644 scripts/lib/wic/canned-wks/etc-overlay.inc create mode > >> 100644 scripts/lib/wic/plugins/source/etc-overlay.py > >> > >> diff --git a/doc/user_manual.md b/doc/user_manual.md > >> index d13a74e..5ec7003 100644 > >> --- a/doc/user_manual.md > >> +++ b/doc/user_manual.md > >> @@ -901,3 +901,29 @@ And build the corresponding image target: > >> ``` > >> bitbake mc:qemuarm64-buster:isar-image-base > >> ``` > >> +## Mount overlay for /etc > >> + > >> +### Motivation > >> + > >> +If a software update solution updates the complete image, the > >> content of `/etc` +is also set to new image content. In this case, > >> the user needs to reconfigure +her system. > >> + > >> +### Approach/Solution > >> + > >> +A possible solution is to create an additional partition, which > >> superimposes +`/etc` with an [overlay > >> file-system](https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt). > >> + +### Example > >> + > >> +Add the following lines to local.conf: > >> +``` > >> +IMAGE_TYPE = "wic-img" > >> +WKS_FILE = "sdimage-efi-overlay" > >> +IMAGE_INSTALL += "etc-overlay" > >> +``` > >> +And build the corresponding image target: > >> + > >> +``` > >> +bitbake mc:qemuamd64-buster:isar-image-base > >> +``` > >> diff --git a/meta-isar/conf/machine/qemuamd64.conf > >> b/meta-isar/conf/machine/qemuamd64.conf index 028b738..d429d96 > >> 100644 --- a/meta-isar/conf/machine/qemuamd64.conf > >> +++ b/meta-isar/conf/machine/qemuamd64.conf > >> @@ -6,11 +6,13 @@ DISTRO_ARCH ?= "amd64" > >> KERNEL_NAME ?= "amd64" > >> > >> IMAGE_TYPE ?= "wic-img" > >> -WKS_FILE ?= "sdimage-efi" > >> +WKS_FILE ?= "sdimage-efi-overlay" > >> IMAGER_INSTALL += "${GRUB_BOOTLOADER_INSTALL}" > >> > >> IMAGE_INSTALL += "sshd-regen-keys" > >> > >> +IMAGE_INSTALL += "etc-overlay" > >> + > >> QEMU_ARCH ?= "x86_64" > >> QEMU_MACHINE ?= "q35" > >> QEMU_CPU ?= "" > >> diff --git a/meta/recipes-support/etc-overlay/etc-overlay_0.1.bb > >> b/meta/recipes-support/etc-overlay/etc-overlay_0.1.bb new file mode > >> 100644 index 0000000..cbed71d > >> --- /dev/null > >> +++ b/meta/recipes-support/etc-overlay/etc-overlay_0.1.bb > >> @@ -0,0 +1,39 @@ > >> +# Create a overlay for /etc to freeze a default configuration > >> +# > >> +# This software is a part of ISAR. > >> +# Copyright (c) Siemens AG, 2020 > >> +# > >> +# SPDX-License-Identifier: MIT > >> + > >> + > >> +DESCRIPTION = "overlay systemd-mount" > >> + > >> +DEBIAN_DEPENDS = "systemd" > >> + > >> +SRC_URI = "file://postinst \ > >> + file://postrm \ > >> + file://etc.mount \ > >> + file://ovl.mount.tmpl \ > >> + file://overlay-parse-etc.service \ > >> + file://etc-hostname.service" > >> + > >> +FS_COMMIT_INTERVAL ?= "20" > >> + > >> +TEMPLATE_VARS += "FS_COMMIT_INTERVAL" > >> +TEMPLATE_FILES += "ovl.mount.tmpl" > >> + > >> +inherit dpkg-raw > >> + > >> +do_install() { > >> + install -m 0755 -d ${D}/ovl > >> + touch ${D}/ovl/.keep > >> + > >> + TARGET=${D}/lib/systemd/system > >> + install -m 0755 -d ${TARGET} > >> + install -m 0644 ${WORKDIR}/etc.mount ${TARGET}/etc.mount > >> + install -m 0644 ${WORKDIR}/ovl.mount ${TARGET}/ovl.mount > >> + install -m 0644 ${WORKDIR}/overlay-parse-etc.service > >> ${TARGET}/overlay-parse-etc.service > >> + install -m 0644 ${WORKDIR}/etc-hostname.service > >> ${TARGET}/etc-hostname.service +} > >> + > >> +addtask do_install after do_transform_template > >> diff --git > >> a/meta/recipes-support/etc-overlay/files/etc-hostname.service > >> b/meta/recipes-support/etc-overlay/files/etc-hostname.service new > >> file mode 100644 index 0000000..2306b9f --- /dev/null > >> +++ b/meta/recipes-support/etc-overlay/files/etc-hostname.service > >> @@ -0,0 +1,14 @@ > >> +[Unit] > >> +Description=set hostname /etc overlay-aware > >> +Before=network-pre.target > >> +Wants=network-pre.target > >> +Requires=etc.mount > >> +After=etc.mount > >> + > >> +[Service] > >> +Type=oneshot > >> +RemainAfterExit=yes > >> +ExecStart=/bin/hostname --boot --file /etc/hostname > >> + > >> +[Install] > >> +WantedBy=basic.target > >> diff --git a/meta/recipes-support/etc-overlay/files/etc.mount > >> b/meta/recipes-support/etc-overlay/files/etc.mount new file mode > >> 100644 index 0000000..59ee0d7 > >> --- /dev/null > >> +++ b/meta/recipes-support/etc-overlay/files/etc.mount > >> @@ -0,0 +1,15 @@ > >> +[Unit] > >> +Description=Overlay-mount /etc > >> +DefaultDependencies=no > >> +Before=local-fs-pre.target > >> +Requires=ovl.mount > >> +After=ovl.mount > >> + > >> +[Mount] > >> +What=overlay > >> +Where=/etc > >> +Type=overlay > >> +Options=noauto,x-systemd.automount,lowerdir=/etc,upperdir=/ovl/etc,workdir=/ovl/.atomic > >> + > >> +[Install] > >> +WantedBy=local-fs-pre.target > >> diff --git > >> a/meta/recipes-support/etc-overlay/files/overlay-parse-etc.service > >> b/meta/recipes-support/etc-overlay/files/overlay-parse-etc.service > >> new file mode 100644 index 0000000..eb5d422 --- /dev/null > >> +++ > >> b/meta/recipes-support/etc-overlay/files/overlay-parse-etc.service > >> @@ -0,0 +1,14 @@ +[Unit] > >> +Description=Reload Configuration from the etc overlay > >> +DefaultDependencies=no > >> +Requires=etc.mount > >> +After=etc.mount > >> +Before=local-fs-pre.target > >> + > >> +[Service] > >> +Type=oneshot > >> +RemainAfterExit=yes > >> +ExecStartPre=!/bin/systemctl daemon-reload > >> +ExecStart=!/bin/systemctl --no-block isolate multi-user.target > >> +[Install] > >> +WantedBy=local-fs-pre.target > >> diff --git a/meta/recipes-support/etc-overlay/files/ovl.mount.tmpl > >> b/meta/recipes-support/etc-overlay/files/ovl.mount.tmpl new file > >> mode 100644 index 0000000..3593c55 > >> --- /dev/null > >> +++ b/meta/recipes-support/etc-overlay/files/ovl.mount.tmpl > >> @@ -0,0 +1,14 @@ > >> +[Unit] > >> +Description=Mount /etc overlay backing store > >> +DefaultDependencies=no > >> +Before=local-fs-pre.target > >> +Before=etc.mount > >> + > >> +[Mount] > >> +What=/dev/disk/by-partlabel/etcovl > >> +Where=/ovl > >> +Type=ext4 > >> +Options=noatime,nodiratime,data=journal,commit=${FS_COMMIT_INTERVAL},nodelalloc > >> + > >> +[Install] > >> +WantedBy=local-fs-pre.target > >> diff --git a/meta/recipes-support/etc-overlay/files/postinst > >> b/meta/recipes-support/etc-overlay/files/postinst new file mode > >> 100755 index 0000000..7a10d68 > >> --- /dev/null > >> +++ b/meta/recipes-support/etc-overlay/files/postinst > >> @@ -0,0 +1,32 @@ > >> +#!/bin/sh > >> +# postinst script for etc-overlay > >> +# > >> +# see: dh_installdeb(1) > >> + > >> +set -e > >> + > >> +case "$1" in > >> + configure) > >> + deb-systemd-helper enable etc.mount || true > >> + deb-systemd-helper enable ovl.mount || true > >> + deb-systemd-helper enable overlay-parse-etc.service || > >> true > >> + deb-systemd-helper enable systemd-remount-fs.service || > >> true > >> + deb-systemd-helper enable etc-hostname.service || true > >> + ;; > >> + > >> + abort-upgrade|abort-remove|abort-deconfigure) > >> + ;; > >> + > >> + *) > >> + echo "postinst called with unknown argument \`$1'" >&2 > >> + exit 1 > >> + ;; > >> +esac > >> + > >> +# dh_installdeb will replace this with shell code automatically > >> +# generated by other debhelper scripts. > >> + > >> +#DEBHELPER# > >> + > >> +exit 0 > >> + > >> diff --git a/meta/recipes-support/etc-overlay/files/postrm > >> b/meta/recipes-support/etc-overlay/files/postrm new file mode > >> 100644 index 0000000..7a3defb > >> --- /dev/null > >> +++ b/meta/recipes-support/etc-overlay/files/postrm > >> @@ -0,0 +1,29 @@ > >> +#!/bin/sh > >> +# postrm script for etc-overlay > >> +# > >> +# see: dh_installdeb(1) > >> + > >> +set -e > >> + > >> + > >> +case "$1" in > >> + > >> purge|remove|upgrade|failed-upgrade|abort-install|abort-upgrade|disappear) > >> + deb-systemd-helper disable etc.mount || true > >> + deb-systemd-helper disable ovl.mount || true > >> + deb-systemd-helper disable overlay-parse-etc.service || > >> true > >> + deb-systemd-helper disable systemd-remount-fs.service || > >> true > >> + deb-systemd-helper disable etc-hostname.service || true > >> + ;; > >> + > >> + *) > >> + echo "postrm called with unknown argument \`$1'" >&2 > >> + exit 1 > >> + ;; > >> +esac > >> + > >> +# dh_installdeb will replace this with shell code automatically > >> +# generated by other debhelper scripts. > >> + > >> +#DEBHELPER# > >> + > >> +exit 0 > >> diff --git a/scripts/lib/wic/canned-wks/etc-overlay.inc > >> b/scripts/lib/wic/canned-wks/etc-overlay.inc new file mode 100644 > >> index 0000000..7b40854 > >> --- /dev/null > >> +++ b/scripts/lib/wic/canned-wks/etc-overlay.inc > >> @@ -0,0 +1,5 @@ > >> +# add a overlay partition to the image > >> + > >> +# overlay partition > >> +part --source etc-overlay --ondisk sda --size 100M --extra-space > >> 128M --overhead-factor 1 --label etcovl --align 1024 --fstype=ext4 > >> + diff --git a/scripts/lib/wic/plugins/source/etc-overlay.py > >> b/scripts/lib/wic/plugins/source/etc-overlay.py new file mode > >> 100644 index 0000000..55189a2 > >> --- /dev/null > >> +++ b/scripts/lib/wic/plugins/source/etc-overlay.py > >> @@ -0,0 +1,84 @@ > >> +# ex:ts=4:sw=4:sts=4:et > >> +# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > >> +# > >> +# Copyright (c) 2014, Intel Corporation. > >> +# Copyright (c) 2018, Siemens AG. > >> +# All rights reserved. > >> +# > >> +# This program is free software; you can redistribute it and/or > >> modify +# it under the terms of the GNU General Public License > >> version 2 as +# published by the Free Software Foundation. > >> +# > >> +# This program is distributed in the hope that it will be useful, > >> +# but WITHOUT ANY WARRANTY; without even the implied warranty of > >> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > >> +# GNU General Public License for more details. > >> +# > >> +# You should have received a copy of the GNU General Public > >> License along +# with this program; if not, write to the Free > >> Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, > >> Boston, MA 02110-1301 USA. +# > >> +# DESCRIPTION > >> +# This implements the 'etc-overlay' source plugin class for 'wic' > >> +# > >> +# AUTHORS > >> +# Tom Zanussi > >> +# Andreas Reichel > >> +# Quirin Gylstorff > >> + > >> + > >> +import logging > >> + > >> +msger = logging.getLogger('wic') > >> + > >> +from wic.pluginbase import SourcePlugin > >> +from wic.utils.misc import exec_cmd,BOOTDD_EXTRA_SPACE > >> + > >> +class EtcOverlayPlugin(SourcePlugin): > >> + """ > >> + Create an overlay file system scheme for etc > >> + """ > >> + > >> + name = 'etc-overlay' > >> + > >> + @classmethod > >> + def do_prepare_partition(cls, part, source_params, creator, > >> cr_workdir, > >> + oe_builddir, deploy_dir, kernel_dir, > >> + rootfs_dir, native_sysroot): > >> + > >> + part_rootfs_dir = "%s/disk/%s.%s" % (cr_workdir, > >> + part.label, > >> + part.lineno) > >> + create_dir_cmd = "install -d %s" % part_rootfs_dir > >> + exec_cmd(create_dir_cmd) > >> + > >> + exec_cmd("install -m 0755 -d %s/etc" % part_rootfs_dir) > >> + exec_cmd("install -m 0755 -d %s/.atomic" % > >> part_rootfs_dir) + > >> + blocks = 16 > >> + extra_blocks = part.get_extra_block_count(blocks) > >> + if extra_blocks < BOOTDD_EXTRA_SPACE: > >> + extra_blocks = BOOTDD_EXTRA_SPACE > >> + blocks += extra_blocks > >> + blocks = blocks + (16 - (blocks % 16)) > >> + > >> + msger.debug("Added %d extra blocks to %s to get to %d > >> total blocks", > >> + extra_blocks, part.mountpoint, blocks) > >> + > >> + # ext4 image, created by mkfs.ext4 > >> + etcovlimg = "%s/%s.%s.img" % (cr_workdir, part.label, > >> part.lineno) > >> + partfs_cmd = "dd if=/dev/zero of=%s bs=512 count=%d" % > >> (etcovlimg, > >> + > >> blocks) > >> + exec_cmd(partfs_cmd) > >> + > >> + partfs_cmd = "mkfs.ext4 %s -d %s" % (etcovlimg, > >> part_rootfs_dir) > >> + exec_cmd(partfs_cmd) > >> + > >> + chmod_cmd = "chmod 644 %s" % etcovlimg > >> + exec_cmd(chmod_cmd) > >> + > >> + du_cmd = "du -Lbks %s" % etcovlimg > >> + etcovlimg_size = int(exec_cmd(du_cmd).split()[0]) > >> + > >> + part.size = etcovlimg_size > >> + part.source_file = etcovlimg > > >