From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6842309179660566528 X-Received: by 2002:a2e:9594:: with SMTP id w20mr12002052ljh.26.1593592173475; Wed, 01 Jul 2020 01:29:33 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:864a:: with SMTP id i10ls414934ljj.8.gmail; Wed, 01 Jul 2020 01:29:32 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxybchkfKHoZhxULgh3slutXA9Gaz6rPwd0Ix9r/ZN9KixzAbczZTf+WF5Sa0VSG2nVgVd8 X-Received: by 2002:a2e:a484:: with SMTP id h4mr12979455lji.468.1593592172630; Wed, 01 Jul 2020 01:29:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1593592172; cv=none; d=google.com; s=arc-20160816; b=sQuYerkxCyriIGwUSgKtU/JJ0Jrza5Pl0t9PQLi3H7zb7QU7j7mVD+7+eXNqmf4aXX nvxF3V3UUVhSPTlPRqXb6I1KRrwXurWK+YgzLuFwTVaEimIH4C8d1DkccA+RcOY8LepR GBKKYt4Wcl7ZMDKWkjzJsC6xnC+y9vRJF3bkccX7iOXqLZzQr99VxapyOxGk+ACCv7gf Fx8gpteuvEJX2GnGM94cCADPTxM4nI0fcJqDWJBMsBhOWtaC13UG4Rd8S1G00WkJA/gn P7hkEa7JDIii5/UpOGC1wgXDwmUj0pJ8yMJtM5PBQ/OCwjunSoADf3bDefhuFnZXoh/H T5yQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=in-reply-to:mime-version:user-agent:date:message-id:organization :autocrypt:from:references:cc:to:subject; bh=GxLJIoIjpYBovgmK+iKW88fECaoLFiiKyHwMgrL+PnE=; b=cd7GKL2P9ZmTguwKrFIM1lmQdvn+pBswAZjFSJzTqgTEBvs0ujM6zx/L8+JuCFg7N7 o9MO7HbJvicC0EUzm955P7O23i13cuaMv6R27lxLGvnCvdMNYwHNm5Nyr5bRJna9Mydp /djSwb3gGe/Eg6x7GKhQg2SOT6wVp1mf03FyLzyE6cXt5UrXDW++gCqT+fxRj3G+qDKS 6/fNKplAfZZ86KKnKmocDg2Vd4RjsfeXk27tQH2zhUcqlM6src0inPV/h6tIxiewLYzv gnqptyykuBIaVp+KAe3/fOmEJ2+INIJMPviDhIDu3YGco1UGzXMOnQh97/olDliEV5yf ZerQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 212.18.0.9 is neither permitted nor denied by best guess record for domain of ch@denx.de) smtp.mailfrom=ch@denx.de Return-Path: Received: from mail-out.m-online.net (mail-out.m-online.net. [212.18.0.9]) by gmr-mx.google.com with ESMTPS id p10si369931ljj.7.2020.07.01.01.29.32 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 01 Jul 2020 01:29:32 -0700 (PDT) Received-SPF: neutral (google.com: 212.18.0.9 is neither permitted nor denied by best guess record for domain of ch@denx.de) client-ip=212.18.0.9; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 212.18.0.9 is neither permitted nor denied by best guess record for domain of ch@denx.de) smtp.mailfrom=ch@denx.de Received: from frontend01.mail.m-online.net (unknown [192.168.8.182]) by mail-out.m-online.net (Postfix) with ESMTP id 49xZCJ0YM2z1qtyB; Wed, 1 Jul 2020 10:29:32 +0200 (CEST) Received: from localhost (dynscan1.mnet-online.de [192.168.6.70]) by mail.m-online.net (Postfix) with ESMTP id 49xZCJ0D6Lz1r56m; Wed, 1 Jul 2020 10:29:32 +0200 (CEST) X-Virus-Scanned: amavisd-new at mnet-online.de Received: from mail.mnet-online.de ([192.168.8.182]) by localhost (dynscan1.mail.m-online.net [192.168.6.70]) (amavisd-new, port 10024) with ESMTP id y1ODwjHQXfRp; Wed, 1 Jul 2020 10:29:29 +0200 (CEST) X-Auth-Info: O2ez/ezimds2So6dQ+ccVhmZTAWpRyK2N931m+qGodc= Received: from [10.0.201.160] (p578adb1c.dip0.t-ipconnect.de [87.138.219.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.mnet-online.de (Postfix) with ESMTPSA; Wed, 1 Jul 2020 10:29:29 +0200 (CEST) Subject: Re: [PATCH] image: Run copy_boot_files after rootfs postprocessing To: Jan Kiszka , Harald Seiler Cc: isar-users@googlegroups.com, Henning Schild References: <20200625153351.3402179-1-hws@denx.de> <20200625184822.236ff069@md1za8fc.ad001.siemens.net> <91ce92c15d267e4836ab4d9de2870bc8e6f6dfa1.camel@denx.de> <5a86e555-416e-d788-2655-003403f1d190@siemens.com> <6bd78ab80bde335821d792046ad2803e4bc9bb57.camel@denx.de> From: Claudius Heine Autocrypt: addr=ch@denx.de; keydata= mQINBFSuWugBEADI18RqDRNsXrWtQBuI60knLMfJ6j26C4ArB5ht4TzTQg9PfTJf1BwKUMYH 1s5qKikAX1mAFm7ca4uxc8uY1AdTFKepNp6ewcRShWPFo1+7DJcTlS1O+cIUyOeo6yrMMI2D DeILzcird0ATI+v7QzO1TmAmEGB28kp9KozZqFKS28p6/GLJ8qCYo50MpacsB0oi8pUeU6+8 KUlKfDDNZvwM+7rVlshOdp0iuAe12IH4gePkXycTm/PvCdqpvm6Uc0FryYh2AzB78Zvzpysb VTdGzFYavdLJJjNifj1XKgeRayR1UG0dMpcUwzLxqRmeRN1Ca7/NVT/NUHm8r80ta6mvjDHj 0vHWfn0CFgfikRtB4tftWmsvYs8YtDtR7X4gaYk5CHhR9iAGU2zElYp2Q417oe1FuXn0eFxG 1SzENtFdUZv64oQSOblJ9ZKslXLBlwCLacSdNK499IEs03UAaTPgh5L/t8G81FDeXduFt129 h6UY6Si2xDYE9wZ90XGag7Uv0wC2LKyOWzEgeOJDIGjQVAMlcr5i6nNa30n/qMTWSf+jahLa 7cr8Mgzw0W9lSAX1CytJJjh5hTHF/atZkl6+vInHJLTtBA2leRF7og2H6PHfUmpJ/A46x71l rODfqjq4/ZblSgqgQCU3rXq8bPwnl7zo/dlyqYUPVAJQ79m9jwARAQABtBtDbGF1ZGl1cyBI ZWluZSA8Y2hAZGVueC5kZT6JAlUEEwEKAD8CGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheA FiEEb/LlnwDGvCgx2GTBEXPLGZgIsVMFAl4pTU4FCQtcJeYACgkQEXPLGZgIsVM+ow//beDC TngJxFrJIgYqHKe93kMBUNjeY9X96nYxELg2dtQQXVeMx+CWsZymP4B9UVXQLGti9ZStKuOB HHlxfVGttwdNeQ1BjjeO9TNJNS0T1jSfz+j/lmSuoCGveojONse2/MOwMNRHtfrkws0JoM1L X3oZkv5JGXDQ51ZVIKXe2+aqIu7oouYpGsDYhxyP2tMhS3J/BHjFDjtQu1H/w64Hq28oa2aY 44prBgz0eCS8V88nlbFPc8K0nZNQYxb0bFcm4VuEHqpSHwbr6Gll9zXVpiOUddv/kbghsz3l u1/7k5Va5ZtRMumhnids5ttmrOIEib+GjpukWGUgZaukF0SNHxhHoAiyklgjEAPMJXYUZjw6 PEUMY+ocYMOgU7uZVLu8rWyCztE0/MW/JawfX/jPAUSRWluA8mP6UNkJDQmuquoaDAv1cQCH h8te/W+Rqa5xYCJCx5B4jEa8Kxmdq6kaWxU2/RXLAHypssiO5Y1XRhDImSjG2SEwJ3nzVpP5 Rd0lDhuoz+UnyidB5Np0tAvQ/4dq4XXxg7/dze0isi58GDtUxMDvtbOnmH2Y/AwxWiM4bC2B gq3JOuG4zAhWFbTG8WHirsvvkzdNXwCyepcR+2jzj1v80k3YqZOYE05POhr0dqeGPFAJJdxl TzwsLPp1z+fsE2nr9jAGd/rLNQ0YCzC5Ag0EVK5a6AEQANbSW8FeAXDMaL26JYiaWriOCB4g zqSIGXPjy1pdtS3dghks2WvADdoUuXBE6ZdbEGl+3QUlXhv167iY1M48oOVCmBnp/ENt1D9s mYB8lby4bVfb5e6eP4VivRUKiU1S2Yvp3en6/Bnts8q6clDezcbWu8lOjzlZEgXbcUPM77r3 3knriTtjnGEGc4Blta/1ED6U00NjqZ3mxpyVT1dmafmMt2Onh6XsYUgCuuAFOkhg0hUB3ems 7NPawURj6PYDYxnbpt/qTJiKbWM6CCkPDHrHesoY/ONvCeGaxzIytaUwXJ/mi3ws36hJXSST Nfyo2W5SskPJvGyn06KXr9YMW9IyZ4AV1pIs2iJ6T1rUXlXpxZ63BMY8Ja8r58Zg5+0cOaBr IAtMCo2aPsfChHEBc3m7XBCfM8yNtvzex8SCVLa/pOVXbKmXKbseInpdtP+3EM1p3R/PjBvK bvsyesk90F6+zsyVu46JVCyqrN3RGGyZCdicFHIPfqzUAnuFIOlUYSDwqvJXMwT5XcZ1jvNR eUvYwqpIJJYumoL3V5SFu434Wu81vMLeFPEccgLW4tbqpckBdbwP8GaPVmgnCWsQhcO9KUDE bnL4nO4lNPgz94dmb579BLsoAH5VVNQONyBEBxeiQBsK3DebwCmQha3qunRc4VUJQlZm/if4 lBdkNMavABEBAAGJAjwEGAEKACYCGwwWIQRv8uWfAMa8KDHYZMERc8sZmAixUwUCXilOFQUJ C1wmrQAKCRARc8sZmAixUyC7EACZIIclsnQc9gLSLxDwBMdEIXmTOEy0tuqJ5MPLNkULH6Sz nPtNBufvkhkKgt56pxTUrxS3ARahLb6AGn8Pl0B2DDtc8SE/L16LCXElssz4VpZ51udKHDzk ijPuaeCevQdTU2Rqdvx26XZre7kQBTXGgvlJAMJlJp0SMfsQ8QdhIHtGLtQmRAzSy6cHNeHX 3HDC7jxrJlJYsmwxbViagYVd9g1D7OE3dCfv9AWqclzbCMLxL0K5QCLLpqiI89dCNKzYyL4F +Cnt98PDa5GG+VXeMB6X6nfWApR20mC1pZP7Tb3XJrEtVkDONejUgQMj9Ao1cPWndma3LAtO aWcenPIPjYQ5Ab38rlE9hEHBfWSC7NriRvCEID7jDmcNG+4j5shKexz7KBoM+Kdr1WjUb3h5 TemrZycE3JGQ4GK46FwRp2O6F4mnrmNdVpnFRT+ilwZ5HOhxKqi8MslDlWmfu6m15JZSvv48 sBWpuXSbJlAKtJkuHVf45gejesDEqG89wE3xGuEnoMsn3+1rpcdLlsXtGuY8fXV4nlPw6Y8e iMuaOyn8qWvxczORDrWk1ZfRlUyV64LDDnbosHoxGLxZ1hf7V6VBtwPqngij2rsMTnCOD11y 3V3xxZAIv7Im43MsUbhZe2andR4UCQFE6NKWG5yHZtCC51APbLwM4ryINad1LA== Organization: Denx Software Engineering Message-ID: Date: Wed, 1 Jul 2020 10:29:25 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.8.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="VSF7kfg0kWUxLbyJPIZbss8moEQQNB3U4" X-TUID: BHY5o+bO5gr6 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --VSF7kfg0kWUxLbyJPIZbss8moEQQNB3U4 Content-Type: multipart/mixed; boundary="GjuWDdJAhxvQUHsWM8475JA2sfD9h0J5a" --GjuWDdJAhxvQUHsWM8475JA2sfD9h0J5a Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable Hi Jan, On 2020-06-29 14:41, Jan Kiszka wrote: > On 29.06.20 14:22, Harald Seiler wrote: >> Hi Jan, >> >> On Fri, 2020-06-26 at 11:12 +0200, Jan Kiszka wrote: >>> On 26.06.20 09:17, Claudius Heine wrote: >>>> Hi Harald, >>>> >>>> On 2020-06-25 19:24, Harald Seiler wrote: >>>>> Hello Henning, >>>>> >>>>> On Thu, 2020-06-25 at 19:02 +0200, Jan Kiszka wrote: >>>>>> On 25.06.20 18:48, [ext] Henning Schild wrote: >>>>>>> Hi Harald, >>>>>>> >>>>>>> can you elaborate on those cases? The postprocessing is hacky, if= >>>>>>> the >>>>>>> problem is coming from your layer you should probably keep this >>>>>>> patch >>>>>>> in you layer. >>>>>> >>>>>> Basically do_generate_image_uuid from >>>>>> https://lore.kernel.org/cip-dev/20200625141015.31719-4-Quirin.Gyls= torff@siemens.com/T/#u, >>>>>> >>>>>> just modeled as post-processing hook, rather than a task. >>>>> >>>>> For reference, this is the exact code: >>>>> >>>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ROOTFS_POSTPROCESS_COMMAND =3D+ "ima= ge_postprocess_generate_uuid" >>>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 image_postprocess_generate_uuid() { >>>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sudo sed -i = '/^IMAGE_UUID=3D.*/d' >>>>> '${IMAGE_ROOTFS}/etc/os-release' >>>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 echo "IMAGE_= UUID=3D\"${IMAGE_UUID}\"" | \ >>>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 sudo tee -a '${IMAGE_ROOTFS}/etc/os-release' >>>>> >>>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sudo -E chro= ot '${ROOTFSDIR}' \ >>>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 update-initramfs -u >>>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 } >>>>> >>>>>> Jan >>>>>> >>>>>>> Maybe you can point out an issue in isar itself, or explain how >>>>>>> you got >>>>>>> into this situation? We can then see if your change is generic >>>>>>> enough >>>>>>> for upstream. You could also provide the error-case from your >>>>>>> layer as >>>>>>> an upstream feature, if that is generic enough. >>>>> >>>>> I think this patch addresses an issue in isar itself.=C2=A0 There i= s no >>>>> reason >>>>> for copy_boot_files() to run before the postprocessing does.=C2=A0 = I've >>>>> checked >>>>> through the git history and the reason this relationship was >>>>> introduced >>>>> was a bigger refactor of the task dependency chain.=C2=A0 It does n= ot >>>>> seem to >>>>> be intentionally this way from what I can tell. >>>>> >>>>> The other way around makes more sense, in my opinion.=C2=A0 As stat= ed in >>>>> the >>>>> commit message, postprocessing might do an update to the initramfs = (as >>>>> seen above) and this change needs to be reflected in the deployed >>>>> initramfs as well, instead of silently only living in the version >>>>> that is >>>>> part of the rootfs. >>>>> >>>>> I also checked all existing postprocessing commands and did not see= >>>>> any >>>>> that assume to be run after the boot files have been deployed. >>>> >>>> Its been a while when I implemented this, but I also thought of the >>>> scenario where someone would like to 'minimize' a image via the root= fs >>>> postprocessing by deleting everything that is not needed, and that >>>> could >>>> possible include the kernel + initramfs, if those are stored somewhe= re >>>> else outside the root file system. So the idea was, IIRC, to move th= e >>>> kernel and initrd to the deploy dir, out of harms way, before >>>> postprocessing does its rootfs manipulation. >>>> >>>> So by ordering the copy_boot_files behind the root fs post processin= g, >>>> you might break other layers that rely on this ordering and have suc= h >>>> 'minimization' procedures, that remove the kernel package and specif= ic >>>> files. >>>> >>>> We don't have such 'minimization' stuff in upstream isar, since it >>>> pretty much breaks apt and dpkg, but if you do image based update, y= ou >>>> might not care. >>> >>> I think the problem with this pattern is elsewhere: We should not >>> install stuff on the rootfs in the first place that shall not end up = in >>> the rootfs. That this copy_boot_files thing depends on the installati= on >>> on the rootfs is actually a bug. It should use the chroot for its wor= k, >>> like the imager does (for the bootloader e.g.). >> >> For kernel and DTB I am totally on your side but I'm not sure how you >> would want to do this for the initramfs.=C2=A0 I think the initramfs s= hould >> definitely be generated from the 'real' rootfs because otherwise you >> might >> get packages from chroot 'polluting' it in unexpected ways.=C2=A0 Also= , >=20 > That is no issue: buildchroot-target and target rootfs are in line. If > not, you have a different issue. I don't think you can rely on that. buildchroot-target is not image specific, its architecture specific, so if you build multiple different images in the same project (same TMPDIR) for the same debian architecture the buildchroot-target will be reused for each on them and possible contains additional packages from other image builds. We should probably implement a similar mechanism as OE, where each recipe gets their own build root fs in isar. Maybe using overlay, or at least hardlinks should be possible. regards, Claudius --GjuWDdJAhxvQUHsWM8475JA2sfD9h0J5a-- --VSF7kfg0kWUxLbyJPIZbss8moEQQNB3U4 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEb/LlnwDGvCgx2GTBEXPLGZgIsVMFAl78SWUACgkQEXPLGZgI sVNXzQ/9HNcTf+we/dIh66a/ra4JCIQCrL3fBK8KaqI5Hpi7+cVNM4gy/8wBzhnK ilEapnENga42drlOPKEP2gOOc2krbFdrrX3fOATxmFoITeVhDV3ooHnIHqSS2nb2 Qj+SXKmRoUzKIDgR3ZrMzm1sT3PagBo+WQ082jfOadETAUsmaX17wKDDmJRKe5xO L6KZTUwJnTQwa3ACYRxRHS/BIqsuwyhGFN/UZxs+9t7OX7XSTveAM6kKE0atT8re MqfChUZw5C4yTRQyaNMvtYULszZC+v7FesAp6t7N9SKGjc8be0VN6+lfT+y8Hibw 4CK3PuMg7wweqoMdZ6/YNtVOdT93enHCVUTuN2n9dxhUoiwP37KtqB0Rwd3KLnet 9ZOhR5QNG2ckMzVZ1oTYkVR54VprRoxXdB5rbSsaeQ3TQym3ULkU5uSzMJ6NTwrK MWM2gD7l+DjQIaFJEFtwtM/gvOTtd88xTQnkmETfKnUvMHg0+awMDPeR0WI26tBy udWxoMoqtQYKb/QMQymL0oiYtQJ/FFkw305yBugfiU72ATDPSobrouc9d190G9TR xfK3YYKCjvQzh7GYF0sas5Wc4SCUGxaJc4QYoELS+Y7Zej9BbAZ7dCfgEEcn1M5/ EE96sNdKuWUW6aBrJEpB49Y3Cb8k2/+LIZ0cI14JaM1ZfTagScU= =g7+U -----END PGP SIGNATURE----- --VSF7kfg0kWUxLbyJPIZbss8moEQQNB3U4--