From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6818448905053667328 X-Received: by 2002:a19:5510:: with SMTP id n16mr16748274lfe.58.1587552990575; Wed, 22 Apr 2020 03:56:30 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:ac2:55ba:: with SMTP id y26ls265788lfg.11.gmail; Wed, 22 Apr 2020 03:56:29 -0700 (PDT) X-Google-Smtp-Source: APiQypJsbv2ydFARsz3hiPthfpHlr+Ot6iiirR/wGVE5oMeBxfaOYjLhEUFhVVXk95U+fedtI1e0 X-Received: by 2002:a19:c1d3:: with SMTP id r202mr16315582lff.216.1587552989789; Wed, 22 Apr 2020 03:56:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1587552989; cv=none; d=google.com; s=arc-20160816; b=JBki7whCZIW7Oxtph2CPrZGq3+0HZP2YJU41C1ycCNzcJgar2++qhlFGYGlj9aU+7b DwTjSzlrYK11Y6EcXmq8i0F00G1r0m+bCIRKU7dZzFQTj5zlxazhV/siE/5t4mLXBpt+ ZqjP8mflT5UHoyHG1HtmKylox0rVE6osvXKTuG2DF0eorN1p7/YqIez5b0lvz6QxPiG5 6MJ2637A7/SzBtehKCAAb50qTGmcdP1AVNdgiUbujKfC0NzMsyN6p5OqwWcvb39voHvp 1wKLclfrxsSI+tXWPeommPmnqAC+8321R4IceyTwMt96xJsjTvbvCIexoJcG7EZDyhq1 py1A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject; bh=4/pKbq3xjnO/EINCEO8gwQMQ7cEpw+vw4LF1JMz55Mg=; b=MzDQS8wZqFzMxbLon6enaspSdI8Kn7QXnIKqcfMxTyW7W9L8sq5klMgYAN4amWKa0B hylAg2FcM38wPnuX0ONx9Ak6AbD/Pa57rcaNvw+EYDRH0Gw1zlkAlgC+MCr6U+RdNYNV DgfuwUrr3UnWqPKyOu19ndXX6wfUqmfz+QeLW1dG/n0DHFR9cV0BUrX/SDXbexBH3LWy dgf7FIWFh5IBhixBc7TTMLBrpNopxy9vRCmJNjtcutJwN0JQxtTDB2SQx+BRzc/H5W2k OhyDodfCf0D/9fB9E+vPzn+iqeElcYflMoanNZ9bECLGvnZOXKNjhci0aF9L07F3FDnJ +SWQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of quirin.gylstorff@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=quirin.gylstorff@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from goliath.siemens.de (goliath.siemens.de. [192.35.17.28]) by gmr-mx.google.com with ESMTPS id a21si439597lfr.4.2020.04.22.03.56.29 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 22 Apr 2020 03:56:29 -0700 (PDT) Received-SPF: pass (google.com: domain of quirin.gylstorff@siemens.com designates 192.35.17.28 as permitted sender) client-ip=192.35.17.28; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of quirin.gylstorff@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=quirin.gylstorff@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by goliath.siemens.de (8.15.2/8.15.2) with ESMTPS id 03MAuSmq027860 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Wed, 22 Apr 2020 12:56:29 +0200 Received: from [139.22.46.195] ([139.22.46.195]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id 03MAuSQb000707; Wed, 22 Apr 2020 12:56:28 +0200 Subject: Re: [PATCH] meta/recipes-support: Mount overlay-fs for /etc To: Henning Schild Cc: isar-users@googlegroups.com, Jan.kiszka@siemens.com References: <20200422082403.7392-1-Quirin.Gylstorff@siemens.com> <20200422115311.750c2ff1@md1za8fc.ad001.siemens.net> From: Gylstorff Quirin Message-ID: <8538b97d-792d-47bd-6f92-3ecea68a4869@siemens.com> Date: Wed, 22 Apr 2020 12:56:28 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.7.0 MIME-Version: 1.0 In-Reply-To: <20200422115311.750c2ff1@md1za8fc.ad001.siemens.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: wD8LyExSfEtu On 4/22/20 11:53 AM, Henning Schild wrote: > Identifying changing content that needs to be persisted is a tough > task, and it goes much further than etc. In fact etc is the least > relevant for an Isar image. Because it usually does not really change > at runtime, maybe except for a first on-first-boot services that > disable themselfs. > What changes outside of isar ... in the worst case because users > postprocess or manually opererate ... can not be covered by isar. > > Debian already protects etc if you choose to update with packages. If > you choose to update a full rootfs and want to persist important stuff > ... /var and /home are way more interesting than /etc. That would need > to be covered as well ... > > I think without a full concept for a read-only rootfs its pointless to > start with the low-hanging etc. > > Henning This feature can be used for read-only fs, but image replacements updates as written in the doc and commit message. And the customer use case was etc but I can easily add /home and /var to the overlay as it is more from the same. Quirin > > On Wed, 22 Apr 2020 10:24:03 +0200 > "Q. Gylstorff" wrote: > >> From: Quirin Gylstorff >> >> Add an overlay fs[1] mount for /etc. This overlay allows an image >> update of root file-system without overwriting the user settings in >> /etc. The overlay mounts during the systemd target >> `local-fs-pre.target.` Due to this target the by systemd generated >> fsck service is not a dependency for the overlay. >> >> [1]: >> https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt >> >> Signed-off-by: Quirin Gylstorff >> --- >> doc/user_manual.md | 26 ++++++ >> meta-isar/conf/machine/qemuamd64.conf | 4 +- >> .../etc-overlay/etc-overlay_0.1.bb | 39 +++++++++ >> .../etc-overlay/files/etc-hostname.service | 14 ++++ >> .../etc-overlay/files/etc.mount | 15 ++++ >> .../files/overlay-parse-etc.service | 14 ++++ >> .../etc-overlay/files/ovl.mount.tmpl | 14 ++++ >> .../etc-overlay/files/postinst | 32 +++++++ >> meta/recipes-support/etc-overlay/files/postrm | 29 +++++++ >> scripts/lib/wic/canned-wks/etc-overlay.inc | 5 ++ >> scripts/lib/wic/plugins/source/etc-overlay.py | 84 >> +++++++++++++++++++ 11 files changed, 275 insertions(+), 1 deletion(-) >> create mode 100644 >> meta/recipes-support/etc-overlay/etc-overlay_0.1.bb create mode >> 100644 meta/recipes-support/etc-overlay/files/etc-hostname.service >> create mode 100644 meta/recipes-support/etc-overlay/files/etc.mount >> create mode 100644 >> meta/recipes-support/etc-overlay/files/overlay-parse-etc.service >> create mode 100644 >> meta/recipes-support/etc-overlay/files/ovl.mount.tmpl create mode >> 100755 meta/recipes-support/etc-overlay/files/postinst create mode >> 100644 meta/recipes-support/etc-overlay/files/postrm create mode >> 100644 scripts/lib/wic/canned-wks/etc-overlay.inc create mode 100644 >> scripts/lib/wic/plugins/source/etc-overlay.py >> >> diff --git a/doc/user_manual.md b/doc/user_manual.md >> index d13a74e..5ec7003 100644 >> --- a/doc/user_manual.md >> +++ b/doc/user_manual.md >> @@ -901,3 +901,29 @@ And build the corresponding image target: >> ``` >> bitbake mc:qemuarm64-buster:isar-image-base >> ``` >> +## Mount overlay for /etc >> + >> +### Motivation >> + >> +If a software update solution updates the complete image, the >> content of `/etc` +is also set to new image content. In this case, >> the user needs to reconfigure +her system. >> + >> +### Approach/Solution >> + >> +A possible solution is to create an additional partition, which >> superimposes +`/etc` with an [overlay >> file-system](https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt). >> + +### Example >> + >> +Add the following lines to local.conf: >> +``` >> +IMAGE_TYPE = "wic-img" >> +WKS_FILE = "sdimage-efi-overlay" >> +IMAGE_INSTALL += "etc-overlay" >> +``` >> +And build the corresponding image target: >> + >> +``` >> +bitbake mc:qemuamd64-buster:isar-image-base >> +``` >> diff --git a/meta-isar/conf/machine/qemuamd64.conf >> b/meta-isar/conf/machine/qemuamd64.conf index 028b738..d429d96 100644 >> --- a/meta-isar/conf/machine/qemuamd64.conf >> +++ b/meta-isar/conf/machine/qemuamd64.conf >> @@ -6,11 +6,13 @@ DISTRO_ARCH ?= "amd64" >> KERNEL_NAME ?= "amd64" >> >> IMAGE_TYPE ?= "wic-img" >> -WKS_FILE ?= "sdimage-efi" >> +WKS_FILE ?= "sdimage-efi-overlay" >> IMAGER_INSTALL += "${GRUB_BOOTLOADER_INSTALL}" >> >> IMAGE_INSTALL += "sshd-regen-keys" >> >> +IMAGE_INSTALL += "etc-overlay" >> + >> QEMU_ARCH ?= "x86_64" >> QEMU_MACHINE ?= "q35" >> QEMU_CPU ?= "" >> diff --git a/meta/recipes-support/etc-overlay/etc-overlay_0.1.bb >> b/meta/recipes-support/etc-overlay/etc-overlay_0.1.bb new file mode >> 100644 index 0000000..cbed71d >> --- /dev/null >> +++ b/meta/recipes-support/etc-overlay/etc-overlay_0.1.bb >> @@ -0,0 +1,39 @@ >> +# Create a overlay for /etc to freeze a default configuration >> +# >> +# This software is a part of ISAR. >> +# Copyright (c) Siemens AG, 2020 >> +# >> +# SPDX-License-Identifier: MIT >> + >> + >> +DESCRIPTION = "overlay systemd-mount" >> + >> +DEBIAN_DEPENDS = "systemd" >> + >> +SRC_URI = "file://postinst \ >> + file://postrm \ >> + file://etc.mount \ >> + file://ovl.mount.tmpl \ >> + file://overlay-parse-etc.service \ >> + file://etc-hostname.service" >> + >> +FS_COMMIT_INTERVAL ?= "20" >> + >> +TEMPLATE_VARS += "FS_COMMIT_INTERVAL" >> +TEMPLATE_FILES += "ovl.mount.tmpl" >> + >> +inherit dpkg-raw >> + >> +do_install() { >> + install -m 0755 -d ${D}/ovl >> + touch ${D}/ovl/.keep >> + >> + TARGET=${D}/lib/systemd/system >> + install -m 0755 -d ${TARGET} >> + install -m 0644 ${WORKDIR}/etc.mount ${TARGET}/etc.mount >> + install -m 0644 ${WORKDIR}/ovl.mount ${TARGET}/ovl.mount >> + install -m 0644 ${WORKDIR}/overlay-parse-etc.service >> ${TARGET}/overlay-parse-etc.service >> + install -m 0644 ${WORKDIR}/etc-hostname.service >> ${TARGET}/etc-hostname.service +} >> + >> +addtask do_install after do_transform_template >> diff --git >> a/meta/recipes-support/etc-overlay/files/etc-hostname.service >> b/meta/recipes-support/etc-overlay/files/etc-hostname.service new >> file mode 100644 index 0000000..2306b9f --- /dev/null >> +++ b/meta/recipes-support/etc-overlay/files/etc-hostname.service >> @@ -0,0 +1,14 @@ >> +[Unit] >> +Description=set hostname /etc overlay-aware >> +Before=network-pre.target >> +Wants=network-pre.target >> +Requires=etc.mount >> +After=etc.mount >> + >> +[Service] >> +Type=oneshot >> +RemainAfterExit=yes >> +ExecStart=/bin/hostname --boot --file /etc/hostname >> + >> +[Install] >> +WantedBy=basic.target >> diff --git a/meta/recipes-support/etc-overlay/files/etc.mount >> b/meta/recipes-support/etc-overlay/files/etc.mount new file mode >> 100644 index 0000000..59ee0d7 >> --- /dev/null >> +++ b/meta/recipes-support/etc-overlay/files/etc.mount >> @@ -0,0 +1,15 @@ >> +[Unit] >> +Description=Overlay-mount /etc >> +DefaultDependencies=no >> +Before=local-fs-pre.target >> +Requires=ovl.mount >> +After=ovl.mount >> + >> +[Mount] >> +What=overlay >> +Where=/etc >> +Type=overlay >> +Options=noauto,x-systemd.automount,lowerdir=/etc,upperdir=/ovl/etc,workdir=/ovl/.atomic >> + >> +[Install] >> +WantedBy=local-fs-pre.target >> diff --git >> a/meta/recipes-support/etc-overlay/files/overlay-parse-etc.service >> b/meta/recipes-support/etc-overlay/files/overlay-parse-etc.service >> new file mode 100644 index 0000000..eb5d422 --- /dev/null >> +++ b/meta/recipes-support/etc-overlay/files/overlay-parse-etc.service >> @@ -0,0 +1,14 @@ >> +[Unit] >> +Description=Reload Configuration from the etc overlay >> +DefaultDependencies=no >> +Requires=etc.mount >> +After=etc.mount >> +Before=local-fs-pre.target >> + >> +[Service] >> +Type=oneshot >> +RemainAfterExit=yes >> +ExecStartPre=!/bin/systemctl daemon-reload >> +ExecStart=!/bin/systemctl --no-block isolate multi-user.target >> +[Install] >> +WantedBy=local-fs-pre.target >> diff --git a/meta/recipes-support/etc-overlay/files/ovl.mount.tmpl >> b/meta/recipes-support/etc-overlay/files/ovl.mount.tmpl new file mode >> 100644 index 0000000..3593c55 >> --- /dev/null >> +++ b/meta/recipes-support/etc-overlay/files/ovl.mount.tmpl >> @@ -0,0 +1,14 @@ >> +[Unit] >> +Description=Mount /etc overlay backing store >> +DefaultDependencies=no >> +Before=local-fs-pre.target >> +Before=etc.mount >> + >> +[Mount] >> +What=/dev/disk/by-partlabel/etcovl >> +Where=/ovl >> +Type=ext4 >> +Options=noatime,nodiratime,data=journal,commit=${FS_COMMIT_INTERVAL},nodelalloc >> + >> +[Install] >> +WantedBy=local-fs-pre.target >> diff --git a/meta/recipes-support/etc-overlay/files/postinst >> b/meta/recipes-support/etc-overlay/files/postinst new file mode 100755 >> index 0000000..7a10d68 >> --- /dev/null >> +++ b/meta/recipes-support/etc-overlay/files/postinst >> @@ -0,0 +1,32 @@ >> +#!/bin/sh >> +# postinst script for etc-overlay >> +# >> +# see: dh_installdeb(1) >> + >> +set -e >> + >> +case "$1" in >> + configure) >> + deb-systemd-helper enable etc.mount || true >> + deb-systemd-helper enable ovl.mount || true >> + deb-systemd-helper enable overlay-parse-etc.service || true >> + deb-systemd-helper enable systemd-remount-fs.service || true >> + deb-systemd-helper enable etc-hostname.service || true >> + ;; >> + >> + abort-upgrade|abort-remove|abort-deconfigure) >> + ;; >> + >> + *) >> + echo "postinst called with unknown argument \`$1'" >&2 >> + exit 1 >> + ;; >> +esac >> + >> +# dh_installdeb will replace this with shell code automatically >> +# generated by other debhelper scripts. >> + >> +#DEBHELPER# >> + >> +exit 0 >> + >> diff --git a/meta/recipes-support/etc-overlay/files/postrm >> b/meta/recipes-support/etc-overlay/files/postrm new file mode 100644 >> index 0000000..7a3defb >> --- /dev/null >> +++ b/meta/recipes-support/etc-overlay/files/postrm >> @@ -0,0 +1,29 @@ >> +#!/bin/sh >> +# postrm script for etc-overlay >> +# >> +# see: dh_installdeb(1) >> + >> +set -e >> + >> + >> +case "$1" in >> + >> purge|remove|upgrade|failed-upgrade|abort-install|abort-upgrade|disappear) >> + deb-systemd-helper disable etc.mount || true >> + deb-systemd-helper disable ovl.mount || true >> + deb-systemd-helper disable overlay-parse-etc.service || true >> + deb-systemd-helper disable systemd-remount-fs.service || true >> + deb-systemd-helper disable etc-hostname.service || true >> + ;; >> + >> + *) >> + echo "postrm called with unknown argument \`$1'" >&2 >> + exit 1 >> + ;; >> +esac >> + >> +# dh_installdeb will replace this with shell code automatically >> +# generated by other debhelper scripts. >> + >> +#DEBHELPER# >> + >> +exit 0 >> diff --git a/scripts/lib/wic/canned-wks/etc-overlay.inc >> b/scripts/lib/wic/canned-wks/etc-overlay.inc new file mode 100644 >> index 0000000..7b40854 >> --- /dev/null >> +++ b/scripts/lib/wic/canned-wks/etc-overlay.inc >> @@ -0,0 +1,5 @@ >> +# add a overlay partition to the image >> + >> +# overlay partition >> +part --source etc-overlay --ondisk sda --size 100M --extra-space >> 128M --overhead-factor 1 --label etcovl --align 1024 --fstype=ext4 + >> diff --git a/scripts/lib/wic/plugins/source/etc-overlay.py >> b/scripts/lib/wic/plugins/source/etc-overlay.py new file mode 100644 >> index 0000000..55189a2 >> --- /dev/null >> +++ b/scripts/lib/wic/plugins/source/etc-overlay.py >> @@ -0,0 +1,84 @@ >> +# ex:ts=4:sw=4:sts=4:et >> +# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> +# >> +# Copyright (c) 2014, Intel Corporation. >> +# Copyright (c) 2018, Siemens AG. >> +# All rights reserved. >> +# >> +# This program is free software; you can redistribute it and/or >> modify +# it under the terms of the GNU General Public License >> version 2 as +# published by the Free Software Foundation. >> +# >> +# This program is distributed in the hope that it will be useful, >> +# but WITHOUT ANY WARRANTY; without even the implied warranty of >> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> +# GNU General Public License for more details. >> +# >> +# You should have received a copy of the GNU General Public License >> along +# with this program; if not, write to the Free Software >> Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. +# >> +# DESCRIPTION >> +# This implements the 'etc-overlay' source plugin class for 'wic' >> +# >> +# AUTHORS >> +# Tom Zanussi >> +# Andreas Reichel >> +# Quirin Gylstorff >> + >> + >> +import logging >> + >> +msger = logging.getLogger('wic') >> + >> +from wic.pluginbase import SourcePlugin >> +from wic.utils.misc import exec_cmd,BOOTDD_EXTRA_SPACE >> + >> +class EtcOverlayPlugin(SourcePlugin): >> + """ >> + Create an overlay file system scheme for etc >> + """ >> + >> + name = 'etc-overlay' >> + >> + @classmethod >> + def do_prepare_partition(cls, part, source_params, creator, >> cr_workdir, >> + oe_builddir, deploy_dir, kernel_dir, >> + rootfs_dir, native_sysroot): >> + >> + part_rootfs_dir = "%s/disk/%s.%s" % (cr_workdir, >> + part.label, >> + part.lineno) >> + create_dir_cmd = "install -d %s" % part_rootfs_dir >> + exec_cmd(create_dir_cmd) >> + >> + exec_cmd("install -m 0755 -d %s/etc" % part_rootfs_dir) >> + exec_cmd("install -m 0755 -d %s/.atomic" % part_rootfs_dir) >> + >> + blocks = 16 >> + extra_blocks = part.get_extra_block_count(blocks) >> + if extra_blocks < BOOTDD_EXTRA_SPACE: >> + extra_blocks = BOOTDD_EXTRA_SPACE >> + blocks += extra_blocks >> + blocks = blocks + (16 - (blocks % 16)) >> + >> + msger.debug("Added %d extra blocks to %s to get to %d total >> blocks", >> + extra_blocks, part.mountpoint, blocks) >> + >> + # ext4 image, created by mkfs.ext4 >> + etcovlimg = "%s/%s.%s.img" % (cr_workdir, part.label, >> part.lineno) >> + partfs_cmd = "dd if=/dev/zero of=%s bs=512 count=%d" % >> (etcovlimg, >> + >> blocks) >> + exec_cmd(partfs_cmd) >> + >> + partfs_cmd = "mkfs.ext4 %s -d %s" % (etcovlimg, >> part_rootfs_dir) >> + exec_cmd(partfs_cmd) >> + >> + chmod_cmd = "chmod 644 %s" % etcovlimg >> + exec_cmd(chmod_cmd) >> + >> + du_cmd = "du -Lbks %s" % etcovlimg >> + etcovlimg_size = int(exec_cmd(du_cmd).split()[0]) >> + >> + part.size = etcovlimg_size >> + part.source_file = etcovlimg > -- Quirin Gylstorff Siemens AG Corporate Technology Research in Digitalization and Automation Smart Embedded Systems CT RDA IOT SES-DE Otto-Hahn-Ring 6 81739 Muenchen, Germany Mobile: +49 173 3746683 mailto:quirin.gylstorff@siemens.com www.siemens.com/ingenuityforlife Siemens Aktiengesellschaft: Chairman of the Supervisory Board: Jim Hagemann Snabe; Managing Board: Joe Kaeser, Chairman, President and Chief Executive Officer; Roland Busch, Lisa Davis, Klaus Helmrich, Cedrik Neike, Michael Sen, Ralf P. Thomas; Registered offices: Berlin and Munich, Germany; Commercial registries: Berlin Charlottenburg, HRB 12300, Munich, HRB 6684; WEEE-Reg.-No. DE 23691322 Important notice: This e-mail and any attachment thereof contain corporate proprietary information. If you have received it by mistake, please notify us immediately by reply e-mail and delete this e-mail and its attachments from your system. Thank you.