From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6842309179660566528 X-Received: by 2002:a1c:7d56:: with SMTP id y83mr15138003wmc.154.1593434492343; Mon, 29 Jun 2020 05:41:32 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:adf:aace:: with SMTP id i14ls7152877wrc.3.gmail; Mon, 29 Jun 2020 05:41:31 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyhbX50s1eZvPihD1tEUcAtZE96WAcXUx0REd9hyeI33dwcXdohEwGDmnclfyQKn+qaRfbh X-Received: by 2002:adf:f60b:: with SMTP id t11mr18029746wrp.249.1593434491687; Mon, 29 Jun 2020 05:41:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1593434491; cv=none; d=google.com; s=arc-20160816; b=rMVkDaeNQ/cr0IZmKi6+tmMZCwwape5z7Huql862clEcIiVTj4ohsebiuPKMomC6d1 SFc+NZOw2xXZQf+z7Rtff3dDhyFvsP9LuH6solEDRXq9kZVkjLPrI25GDM3OI0eIemh2 7Nj7bGBUq+0ya5uxmDmvIYg/nr0Dcr5TMXXJ6DW/KFykbriDmPbumVU9uNSdSJzsiz63 4xwWVyMdP1q5Rf1iHsiPCCMoP3xW20aCJgc9Q2tCSJcXDTCc6Axf/GU8uZwtGyFT59vP UAchmapKpATezyNjkST2zZhHIT2eJTGXG5h/gFlkqBLc+pC/YxoTHfh2Rirv4igxhFgU eKYg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject; bh=7aYa1TmeZ7KhHKmQVFSLpV6ewocrUHMo9wa5jgkUUkM=; b=TSxFVwiX0wkaz25O7OF/coiXB7Srsneok5Y7uWuL83MBwij6/Difzr9f23Niy+9GFf wR29TBIe3vMBwcg4ciNYvicEdMM5ohad5ATveVpewHxiVrAkaq/Y8gNIbu31EdW+/sNe mwQwseuaNyBdV8PGjgB9Sk+dLP8X/Q6DpBKUaYWxyDC97YqgW3jC5DTIm/bHaKHKz93l jwPnDvnJ3NDpP/y/4dNiCO51nnRA2gHkzyke4pAlqtor1RG9+yNM6TXaqKtFfEAd3vWH n/kB2BCelsn0DVq7YG3/49O9yr/OAeTusGE743gk/RSELDFDZOykADdptbA1wmdLREYB wgbQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from goliath.siemens.de (goliath.siemens.de. [192.35.17.28]) by gmr-mx.google.com with ESMTPS id o201si900341wme.1.2020.06.29.05.41.31 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Jun 2020 05:41:31 -0700 (PDT) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.28 as permitted sender) client-ip=192.35.17.28; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by goliath.siemens.de (8.15.2/8.15.2) with ESMTPS id 05TCfUn7014462 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 29 Jun 2020 14:41:30 +0200 Received: from [167.87.50.192] ([167.87.50.192]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id 05TCfTW2011146; Mon, 29 Jun 2020 14:41:30 +0200 Subject: Re: [PATCH] image: Run copy_boot_files after rootfs postprocessing To: Harald Seiler , Claudius Heine Cc: isar-users@googlegroups.com, Henning Schild References: <20200625153351.3402179-1-hws@denx.de> <20200625184822.236ff069@md1za8fc.ad001.siemens.net> <91ce92c15d267e4836ab4d9de2870bc8e6f6dfa1.camel@denx.de> <5a86e555-416e-d788-2655-003403f1d190@siemens.com> <6bd78ab80bde335821d792046ad2803e4bc9bb57.camel@denx.de> From: Jan Kiszka Message-ID: Date: Mon, 29 Jun 2020 14:41:29 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 In-Reply-To: <6bd78ab80bde335821d792046ad2803e4bc9bb57.camel@denx.de> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: Be3vd2oo4uuY On 29.06.20 14:22, Harald Seiler wrote: > Hi Jan, > > On Fri, 2020-06-26 at 11:12 +0200, Jan Kiszka wrote: >> On 26.06.20 09:17, Claudius Heine wrote: >>> Hi Harald, >>> >>> On 2020-06-25 19:24, Harald Seiler wrote: >>>> Hello Henning, >>>> >>>> On Thu, 2020-06-25 at 19:02 +0200, Jan Kiszka wrote: >>>>> On 25.06.20 18:48, [ext] Henning Schild wrote: >>>>>> Hi Harald, >>>>>> >>>>>> can you elaborate on those cases? The postprocessing is hacky, if the >>>>>> problem is coming from your layer you should probably keep this patch >>>>>> in you layer. >>>>> >>>>> Basically do_generate_image_uuid from >>>>> https://lore.kernel.org/cip-dev/20200625141015.31719-4-Quirin.Gylstorff@siemens.com/T/#u, >>>>> just modeled as post-processing hook, rather than a task. >>>> >>>> For reference, this is the exact code: >>>> >>>> ROOTFS_POSTPROCESS_COMMAND =+ "image_postprocess_generate_uuid" >>>> image_postprocess_generate_uuid() { >>>> sudo sed -i '/^IMAGE_UUID=.*/d' '${IMAGE_ROOTFS}/etc/os-release' >>>> echo "IMAGE_UUID=\"${IMAGE_UUID}\"" | \ >>>> sudo tee -a '${IMAGE_ROOTFS}/etc/os-release' >>>> >>>> sudo -E chroot '${ROOTFSDIR}' \ >>>> update-initramfs -u >>>> } >>>> >>>>> Jan >>>>> >>>>>> Maybe you can point out an issue in isar itself, or explain how you got >>>>>> into this situation? We can then see if your change is generic enough >>>>>> for upstream. You could also provide the error-case from your layer as >>>>>> an upstream feature, if that is generic enough. >>>> >>>> I think this patch addresses an issue in isar itself. There is no reason >>>> for copy_boot_files() to run before the postprocessing does. I've checked >>>> through the git history and the reason this relationship was introduced >>>> was a bigger refactor of the task dependency chain. It does not seem to >>>> be intentionally this way from what I can tell. >>>> >>>> The other way around makes more sense, in my opinion. As stated in the >>>> commit message, postprocessing might do an update to the initramfs (as >>>> seen above) and this change needs to be reflected in the deployed >>>> initramfs as well, instead of silently only living in the version that is >>>> part of the rootfs. >>>> >>>> I also checked all existing postprocessing commands and did not see any >>>> that assume to be run after the boot files have been deployed. >>> >>> Its been a while when I implemented this, but I also thought of the >>> scenario where someone would like to 'minimize' a image via the root fs >>> postprocessing by deleting everything that is not needed, and that could >>> possible include the kernel + initramfs, if those are stored somewhere >>> else outside the root file system. So the idea was, IIRC, to move the >>> kernel and initrd to the deploy dir, out of harms way, before >>> postprocessing does its rootfs manipulation. >>> >>> So by ordering the copy_boot_files behind the root fs post processing, >>> you might break other layers that rely on this ordering and have such >>> 'minimization' procedures, that remove the kernel package and specific >>> files. >>> >>> We don't have such 'minimization' stuff in upstream isar, since it >>> pretty much breaks apt and dpkg, but if you do image based update, you >>> might not care. >> >> I think the problem with this pattern is elsewhere: We should not >> install stuff on the rootfs in the first place that shall not end up in >> the rootfs. That this copy_boot_files thing depends on the installation >> on the rootfs is actually a bug. It should use the chroot for its work, >> like the imager does (for the bootloader e.g.). > > For kernel and DTB I am totally on your side but I'm not sure how you > would want to do this for the initramfs. I think the initramfs should > definitely be generated from the 'real' rootfs because otherwise you might > get packages from chroot 'polluting' it in unexpected ways. Also, That is no issue: buildchroot-target and target rootfs are in line. If not, you have a different issue. > considering the IMAGE_UUID use-case in particular, how would you get the > ID from the rootfs into the chroot for the hook to pick it up? Not sure > whether there is a clean solution for this ... > E.g. by generating the UUID at bitbake level and injecting it into the images - rather than doing that inside one of the images. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux