From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6842309179660566528 X-Received: by 2002:a19:6a02:: with SMTP id u2mr9325374lfu.9.1593437162412; Mon, 29 Jun 2020 06:26:02 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:ac2:5f95:: with SMTP id r21ls2303360lfe.3.gmail; Mon, 29 Jun 2020 06:26:01 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwdTEG5nJJCC9I2b0zirlSOC2Y/z7wTMbxcQ+dsRSiYWtnk4H4wEAMZd/AyhTHpZzOSjlVe X-Received: by 2002:a19:c214:: with SMTP id l20mr9124118lfc.56.1593437161527; Mon, 29 Jun 2020 06:26:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1593437161; cv=none; d=google.com; s=arc-20160816; b=0AVsyB0HMZCwVtBdL7AivVrJFor3nzSs1CEDHIJ2Vr5cxZ3gVYAqvvBLbfM1BEiPdK 4D3LQ+nGtVzSLiUdZHhfndeT8RQFU1vhvGOvAiug7hqY8wOl0UlbJcCdYyCNr0fv9z7r EXo9hsgEHpiv6TG+16HlJABSfnixFt9qkQhrky3PI+QZo0TpEidtkiO0TMloxsqYMxtW TptaAdHtZL45FEsILjaMX/KuOVvjy7n1J74FiniBM/k9RUlAR8/WSgjNDKp1RAbAVjQ1 bY447rzR+9K760Ofn+8i7XKctfDZoGDW6M0nH2pCKazaEa2JfCgjzFECWSGZfLXD1W4G BV/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject; bh=nIcYoEc3AXVNBDI/Dus98l8vi3DL7Ey9Gd4VPZkx7lY=; b=Xon3L0SD4IxhFDrhNG02zLv+ze8xq3BRH8ZKJ6r7wwDcGskSfLFVVkSL88VHRsdiKo 7f1zUPwEm93k3qC4PLhKP4q9c09ptJIJ6sZrYFmJDSViTC7Q2VSzln+BTHkev6pIPBOy OUKGrR4f3Xav1YDzGOrfbTGPkomavvYqQVDAJXvTJHWtAPaHm/YxZTGTz/7l2PFu08/k Nq1paP/Ikc1EFJfjHRKjX45rvibgcjCEg/LlCY/IkqYJhFCNDI+EMI8bdq8u6gWIaSjS wAKAlsLneFg9vjeMl6F7OfllkT+WbNQE2AvNIRx1MDqw2MPm9CxVTOr9j++lLA00SawY W6Jw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id d9si993ljj.1.2020.06.29.06.26.01 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Jun 2020 06:26:01 -0700 (PDT) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id 05TDQ0cl027465 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 29 Jun 2020 15:26:00 +0200 Received: from [167.87.50.192] ([167.87.50.192]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id 05TDPxRt017492; Mon, 29 Jun 2020 15:25:59 +0200 Subject: Re: [PATCH] image: Run copy_boot_files after rootfs postprocessing To: Henning Schild Cc: Harald Seiler , Claudius Heine , isar-users@googlegroups.com References: <20200625153351.3402179-1-hws@denx.de> <20200625184822.236ff069@md1za8fc.ad001.siemens.net> <91ce92c15d267e4836ab4d9de2870bc8e6f6dfa1.camel@denx.de> <5a86e555-416e-d788-2655-003403f1d190@siemens.com> <6bd78ab80bde335821d792046ad2803e4bc9bb57.camel@denx.de> <20200629145528.37c2c8d8@md1za8fc.ad001.siemens.net> From: Jan Kiszka Message-ID: Date: Mon, 29 Jun 2020 15:25:59 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 In-Reply-To: <20200629145528.37c2c8d8@md1za8fc.ad001.siemens.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: W58M3umjriCP On 29.06.20 14:55, Henning Schild wrote: > On Mon, 29 Jun 2020 14:41:29 +0200 > Jan Kiszka wrote: > >> On 29.06.20 14:22, Harald Seiler wrote: >>> Hi Jan, >>> >>> On Fri, 2020-06-26 at 11:12 +0200, Jan Kiszka wrote: >>>> On 26.06.20 09:17, Claudius Heine wrote: >>>>> Hi Harald, >>>>> >>>>> On 2020-06-25 19:24, Harald Seiler wrote: >>>>>> Hello Henning, >>>>>> >>>>>> On Thu, 2020-06-25 at 19:02 +0200, Jan Kiszka wrote: >>>>>>> On 25.06.20 18:48, [ext] Henning Schild wrote: >>>>>>>> Hi Harald, >>>>>>>> >>>>>>>> can you elaborate on those cases? The postprocessing is hacky, >>>>>>>> if the problem is coming from your layer you should probably >>>>>>>> keep this patch in you layer. >>>>>>> >>>>>>> Basically do_generate_image_uuid from >>>>>>> https://lore.kernel.org/cip-dev/20200625141015.31719-4-Quirin.Gylstorff@siemens.com/T/#u, >>>>>>> just modeled as post-processing hook, rather than a task. >>>>>> >>>>>> For reference, this is the exact code: >>>>>> >>>>>> ROOTFS_POSTPROCESS_COMMAND =+ >>>>>> "image_postprocess_generate_uuid" >>>>>> image_postprocess_generate_uuid() { sudo sed -i >>>>>> '/^IMAGE_UUID=.*/d' '${IMAGE_ROOTFS}/etc/os-release' echo >>>>>> "IMAGE_UUID=\"${IMAGE_UUID}\"" | \ sudo tee -a >>>>>> '${IMAGE_ROOTFS}/etc/os-release' >>>>>> >>>>>> sudo -E chroot '${ROOTFSDIR}' \ >>>>>> update-initramfs -u >>>>>> } >>>>>> >>>>>>> Jan >>>>>>> >>>>>>>> Maybe you can point out an issue in isar itself, or explain >>>>>>>> how you got into this situation? We can then see if your >>>>>>>> change is generic enough for upstream. You could also provide >>>>>>>> the error-case from your layer as an upstream feature, if that >>>>>>>> is generic enough. >>>>>> >>>>>> I think this patch addresses an issue in isar itself. There is >>>>>> no reason for copy_boot_files() to run before the postprocessing >>>>>> does. I've checked through the git history and the reason this >>>>>> relationship was introduced was a bigger refactor of the task >>>>>> dependency chain. It does not seem to be intentionally this way >>>>>> from what I can tell. >>>>>> >>>>>> The other way around makes more sense, in my opinion. As stated >>>>>> in the commit message, postprocessing might do an update to the >>>>>> initramfs (as seen above) and this change needs to be reflected >>>>>> in the deployed initramfs as well, instead of silently only >>>>>> living in the version that is part of the rootfs. >>>>>> >>>>>> I also checked all existing postprocessing commands and did not >>>>>> see any that assume to be run after the boot files have been >>>>>> deployed. >>>>> >>>>> Its been a while when I implemented this, but I also thought of >>>>> the scenario where someone would like to 'minimize' a image via >>>>> the root fs postprocessing by deleting everything that is not >>>>> needed, and that could possible include the kernel + initramfs, >>>>> if those are stored somewhere else outside the root file system. >>>>> So the idea was, IIRC, to move the kernel and initrd to the >>>>> deploy dir, out of harms way, before postprocessing does its >>>>> rootfs manipulation. >>>>> >>>>> So by ordering the copy_boot_files behind the root fs post >>>>> processing, you might break other layers that rely on this >>>>> ordering and have such 'minimization' procedures, that remove the >>>>> kernel package and specific files. >>>>> >>>>> We don't have such 'minimization' stuff in upstream isar, since it >>>>> pretty much breaks apt and dpkg, but if you do image based >>>>> update, you might not care. >>>> >>>> I think the problem with this pattern is elsewhere: We should not >>>> install stuff on the rootfs in the first place that shall not end >>>> up in the rootfs. That this copy_boot_files thing depends on the >>>> installation on the rootfs is actually a bug. It should use the >>>> chroot for its work, like the imager does (for the bootloader >>>> e.g.). >>> >>> For kernel and DTB I am totally on your side but I'm not sure how >>> you would want to do this for the initramfs. I think the initramfs >>> should definitely be generated from the 'real' rootfs because >>> otherwise you might get packages from chroot 'polluting' it in >>> unexpected ways. Also, >> >> That is no issue: buildchroot-target and target rootfs are in line. >> If not, you have a different issue. >> >>> considering the IMAGE_UUID use-case in particular, how would you >>> get the ID from the rootfs into the chroot for the hook to pick it >>> up? Not sure whether there is a clean solution for this ... >>> >> >> E.g. by generating the UUID at bitbake level and injecting it into >> the images - rather than doing that inside one of the images. > > I anyway do not get why that UUID is needed in the first place. Which > partition to take as root partition can be decided with a partition > label. You just have to be careful with swupdate, it destroy those > labels when flashing whole partitions. We need to identify the *filesystem* instance, not the partition, while booting. A new version of the filesystem may be in partition A or partition B, depending on how often the system was already updated. I'm literally just now painting this slide for my talk tomorrow... Jan > Even if you decide to use an UUID, the bootloader should know that and > pass that arg. > > ... root=$(get_part_uuid(initrd-file)) ... > > Might require improving bootloaders ;) or writing bootloader configs > instead of initrds in postrprocess ... > > You can also put your static desired UUID in your wks-file and a > postinst of a customization-package. But that might be only for > recipes/layers that are not meant for sharing > > Henning > >> Jan >> > -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux