From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7187242631035879424 X-Received: by 2002:a50:ec93:0:b0:482:9a72:131e with SMTP id e19-20020a50ec93000000b004829a72131emr5887378edr.160.1673441287462; Wed, 11 Jan 2023 04:48:07 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6402:26ca:b0:45c:bfd9:fb45 with SMTP id x10-20020a05640226ca00b0045cbfd9fb45ls456936edd.3.-pod-prod-gmail; Wed, 11 Jan 2023 04:48:06 -0800 (PST) X-Google-Smtp-Source: AMrXdXs67KIFhWHe+bTzzLJy7Vf3E06BLMIRDSzI1w0xNvwv7sSP3l0FOcPZtNmrxFl7TycyG2WS X-Received: by 2002:a05:6402:c4c:b0:48f:acd1:8da2 with SMTP id cs12-20020a0564020c4c00b0048facd18da2mr23838196edb.27.1673441286340; Wed, 11 Jan 2023 04:48:06 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1673441286; cv=pass; d=google.com; s=arc-20160816; b=PBmIpvrk78PPKUvOambpfrmh2JgkcxE59MojYM0yD4WJe1NGSe2THDZm1oHoSnaR/I HhVtEd4ZutuIAzDSa6oGiPfa6UO7S9zNIYLXlr3c4HwxjiUc1wcS+WTRwNMiApxS3dm+ ov7bsVD3oSs8GFmaP2T72xOZgQb9auZwn2OKRNKU6uHxgzPTFKcVepDx0vH5gddv0SRP g0SaVypPw0lToV0ms0V8uNtPRIlaGlkgTwD4fdigNTzM6QAuAeZTIhJTxO1ivuNOPpyc EMhlQHMQq68mBmEOTaJXQseru36pDCGiNx28l4wkwBMqOb40Yrw4RboCFJJuFzxhxmrD rvMQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:content-transfer-encoding:references:in-reply-to :message-id:subject:cc:to:from:date:dkim-signature; bh=zOLp3DtotllNXC/LBUag7gjk231HgdO0W3J7mNSFvrg=; b=ESPFWS4yA975LQPxMgaT/wb+6EnJAngOjUq/TzCS6LylN6IU73ALjMjnj1OJwTHX/W yQB0U+u+914E28bm1httRMP5ke6gfA+ADlXVcBREMt/kaBlCUzgCeXq0sQqW1wV44Se/ E2VQPHq3Qb4LdS+mHCL2ci6qgaOT33CpBGkz/vv/I7p2C5QT3hq2jGd6N21qQJwSA8jH nrtu+KXtqijspoyK+3XZ6y0Izbrg1Eu5enW5hQyDoa90Gf3nhoI8vB3A5xCqGVO99N5k T+ONbNGMrhoy6jYoy2HUOpNceqmiKeZxz6uAqlvMY5genuIHxojKdOmt8lJGZ+aCL/wm hQoQ== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=V8ITkTI3; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of henning.schild@siemens.com designates 40.107.8.88 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from EUR04-VI1-obe.outbound.protection.outlook.com (mail-vi1eur04on2088.outbound.protection.outlook.com. [40.107.8.88]) by gmr-mx.google.com with ESMTPS id k20-20020a05640212d400b0047014e8771fsi592767edx.3.2023.01.11.04.48.06 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 11 Jan 2023 04:48:06 -0800 (PST) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 40.107.8.88 as permitted sender) client-ip=40.107.8.88; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=V8ITkTI3; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of henning.schild@siemens.com designates 40.107.8.88 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ldhpGp9ExlIstv2b4gXCUC5/j3EhkHz0InVmU2ViQS1mHURA5Q+alrWkR2mQRgfzLLJrMLEjX8jZ3Wc65x9QPRlMvNHCDxNKcWSM3ByTtjzeN95c7kfcX+ZEX/v4QqBfy7AgiTdF3RG5sX/uGF8ZL0w9r10pzipx74bTiaO1BVb+DM3dqLw9m7dbPUd63tW2m3I6YTpT+ta4lkUGaDUfHGvgw6gmBJU9h1nkVjFlrNNlBYDywlkuYD9qWyiSUsVC1LO6wUb76dwsQ8pILyO39GOCIiaRbbtniADmkjIT0xrp9tfAfvdy36W91V2s2JipKdUw7C4K+09Z/LhX+O97Eg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zOLp3DtotllNXC/LBUag7gjk231HgdO0W3J7mNSFvrg=; b=j7sUBA9jUCQXwmutFoU0Zctsbyu1VdesoKzYPOkWA95fm8d9pIsqudkRyf5AbV+vvhCZpNVeJfkMqVDyo3DJFA7LCHfuv5gD1nYxqzW5AOgU/bsKf5+89ItHuKKPN+yOSqF2lI9LxvwKy9ozm5mnHffvIbwuXtG9YOFxFBl1pbIcPqLOArk6kNQBg0ARoRBUcOnsyzfFwrgWBea6SwrjPnMCykn7NG4HfV7wGnaYJRqHrOKmcFOADBozxKCBaAhCU2aYqXQBrf6uCLYQ6HcT/jJ04DpmDLna7NPUu2DWuqSb6Klfxn+GPNiF5G8QPCe3qJ7ejRn7SxRNEkW0AHNCEA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=siemens.com; dmarc=pass action=none header.from=siemens.com; dkim=pass header.d=siemens.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zOLp3DtotllNXC/LBUag7gjk231HgdO0W3J7mNSFvrg=; b=V8ITkTI3ZcKC5mPAOPau9a3Brr+i9hL42lOZ3GtoeWk7v4vovY2u37KRK29W2CjII6prSVlS4L6HpGDk/TzyzKerReejcjf2Y3XkMFEL6YYP02vMf0cLQEwHW0wQcfxEF5fr19T0WFSBY1bT6czgEsbRdyo/R7DGdq2pHgXrDoVzBgw0+pwSSN6aiFyIwPWF5pYKeWovGodJHmHUA9f242DTAPmXJQBzz8kfmwknRm87dmLwmXUY+5ixyGIxeB72yKa9+y0U3PNFKPN+VtVodUeXgpiGPSsaOh9TEcSUCMnAuofRHKxd5XqLdLd2xpB/a8w7Co10bOvTFvui2t+L8g== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=siemens.com; Received: from PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:102:269::8) by AM7PR10MB3176.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:101::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5944.19; Wed, 11 Jan 2023 12:48:05 +0000 Received: from PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM ([fe80::bdf0:fdeb:f955:bc79]) by PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM ([fe80::bdf0:fdeb:f955:bc79%4]) with mapi id 15.20.6002.012; Wed, 11 Jan 2023 12:48:05 +0000 Date: Wed, 11 Jan 2023 13:47:56 +0100 From: Henning Schild To: "Moessbauer, Felix (T CED INW-CN)" Cc: "Bovensiepen, Daniel (bovi) (T CED INW-CN)" , "isar-users@googlegroups.com" , "Kiszka, Jan (T CED)" , "venkata.pyla@toshiba-tsip.com" Subject: Re: [PATCH 03/11] rootfs postprocess: clean python cache Message-ID: <20230111134756.77c9564a@md1za8fc.ad001.siemens.net> In-Reply-To: <9e2bda0ce8c1c4cebdb57f9a40cc4db91d0a7d83.camel@siemens.com> References: <20230111041140.3460393-1-felix.moessbauer@siemens.com> <20230111041140.3460393-4-felix.moessbauer@siemens.com> <20230111090626.7f2bc56c@md1za8fc.ad001.siemens.net> <9e2bda0ce8c1c4cebdb57f9a40cc4db91d0a7d83.camel@siemens.com> X-Mailer: Claws Mail 4.1.0 (GTK 3.24.35; x86_64-pc-linux-gnu) Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-ClientProxiedBy: CH0PR03CA0194.namprd03.prod.outlook.com (2603:10b6:610:e4::19) To PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:102:269::8) Return-Path: henning.schild@siemens.com MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PA4PR10MB5780:EE_|AM7PR10MB3176:EE_ X-MS-Office365-Filtering-Correlation-Id: ec4de3ea-da04-4169-c4bb-08daf3d21493 X-LD-Processed: 38ae3bcd-9579-4fd4-adda-b42e1495d55a,ExtAddr X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 2YpevqnoSoTXYie1D1LTXAT0cotiBLMijdrsCZeMolmbwSWERcrqlB++BdaDo3Pbirw0jK8m14NkW71dleu/fEcKLl6IiYA0xMdhD+ir2/tQpnzgjgNK2NnniF1FxyMBzdO3kpo/wqOUzYM3bb4wNYH28t9dCWQ1ANEhIsgS2FxYEBZSBk1fvzPTtjGyganfpn1UAdvtL7+61gNukVhXZ2bU7Zk9z6iHq/QVgNvR+hbhW4M1G3UEFrwECEp+RSUTirPjMCwwgC+l0cC77+7WmV2Dpn2KzaWx1gCMR0DPxYfJq2cu9pI37ePR2JHC+EJHTeb7oZU3NXsMf7RPCkMrDVKPxiAj7BP8gUTZM8u4sFlgyhyUHXen9iI7maVdrO8w2NB6OrWwrmUHgj6KiH3WJW/N4lH//3glRlgVGs10uVovCaOxp5xUNP3tUJTIYpsVtAwK4VYniRp3kMagiNtomLuVt9TE7F4H19T40u0e46vGJwz82EZFMW96UExABbQqxPbBZ/+vgYKhMsNiSvwMkiOxXbqithDIamySzBJ4vTkbLWp5BKArI8BaCpNj8Y0wBlyJIgYrrCTMryg4OIVTFWpyA2aZVfrGqr+LIREpeNG+86zVCsSAvGVZdqVTCUL3RwzRQpTYmalOfZok1mOGJhEcyvf3pYfuhHrYXUvj4wM= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230022)(4636009)(346002)(396003)(136003)(39860400002)(376002)(366004)(451199015)(478600001)(86362001)(1076003)(82960400001)(6486002)(54906003)(316002)(6636002)(8936002)(966005)(38100700002)(2906002)(4326008)(6506007)(44832011)(5660300002)(41300700001)(186003)(8676002)(66476007)(66556008)(83380400001)(6512007)(66946007)(26005)(6666004)(9686003)(6862004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?Lp2laFJDhA78li57HZQCx9KT4pe305u8M0kXh0uMwp35SWz/4EdTCRc6dotb?= =?us-ascii?Q?HddbsKxYpsZt2l5kM9p9JygR0wzdayflqxsU1J/fYKdLIPLqc4Pa4TpXF5R0?= =?us-ascii?Q?hLJMa2XOD+7AW27sHJ4HhTW0AYU1yQpHW4Pc0D7Di9yG/vFS8X9j9Az7Jhew?= =?us-ascii?Q?nbgMO/QOtuBZt+g0RsO4uLosVoHdzVlu6cphCnPmH9hAVR3EXaGfSsdmTJsb?= =?us-ascii?Q?cbvuysW/xE25zH6bniyPg0cYyrQwuX2MI+E/DkNERJfTjd2EXAoApKY5WIm1?= =?us-ascii?Q?8ksSKK0oJWbJ0M1QP8JtZdbkZVY+nZ0E1Yllaa8q7IC02UAe4GEKx/L2wyBI?= =?us-ascii?Q?UBFVtqGQ14f31pkRvc8+BXWT4xG2f37mhXxkmNdlClpny61/aa2NVTHmmx7F?= =?us-ascii?Q?zQIIE9NoPmGEsRvnxNiRNe1fwTpbk6hcXAtSzOWR5uHKDBrR3ewKqtP+du51?= =?us-ascii?Q?c0b+z1LpZ4NJrgmhMmUGthArtYlcChTkquAHYcd68sov/MqQPQHYu4qKZgxq?= =?us-ascii?Q?ww7EfJ76b2re7LtmWvKWAmOwUYarTm/RdIY7cA/GlPhWV74WcCs0T4dk0J24?= =?us-ascii?Q?pVY+mbK3M85skYXwBtEbr4QuU0vzC8p688cRdxfiKCUv+4IIJuXMZ9aoMp/y?= =?us-ascii?Q?oxmtf92lp+ch0sD1M5wxHp/JJmSybUPJyuYQK3U/DYJ0S6rG7MBxtZL7XAoh?= =?us-ascii?Q?X+h8bBFTdlqs74bBM8Ew+7YcvwlFSil09+Tw4jt+ZHnUn2lmPdNoa6/K5nzl?= =?us-ascii?Q?L50kKfNXCOGkY+mkJ5c2iV3Mfy9KXmrJzHTGa2N3uaPzm1YxNZyx+iTvIAl5?= =?us-ascii?Q?0jHbVsriFXkQY9vubvbYbPcixQzxYot3PvLcGkGsxOvQizN6u5a8bmkTbPY7?= =?us-ascii?Q?2zcVFc+cIOFIGam3VB6kFFxlahB4UNypGUo9HM5xew6tCzNb6shbniEQiLiK?= =?us-ascii?Q?APNkyDubm3y1W83b3RQXUIypGdPSWXBo5FjpYThQxcggfWxVYLvmeWbMKo6+?= =?us-ascii?Q?qZ17Spfc1NH4RnK49S1A1r3aX0A9W8sJoJVHe/ukLZmc/z+x6d5enFrefF9H?= =?us-ascii?Q?qUy1CRI//jAJdja+8qHSgcU36dzdBA+6YVaCCtxYdg+AttNeSxECxs6FOwRD?= =?us-ascii?Q?HjqLKsZO14s6Mijs6aEw5dwrMquhNo12BH+j1zvF+9rf4epiy0tLcV8DEA+U?= =?us-ascii?Q?8oo31rF8s+WHFfx3MW8LdAssQ3RBoVnkqyUyqIliXTLZZ7JwkA0ksroKQDhu?= =?us-ascii?Q?BW0N84UuqS71Ryq/DQSxC6YKFKRGJxGKGCCmJh+X8pmbH0AYP8OR/LrQDjic?= =?us-ascii?Q?Hsx1C6AVCg4ISghZdyrj5KBgKLA7gutxIkLv5PIMAQwStDe2I3bQ8bTF1Vrb?= =?us-ascii?Q?Pp8Qroo4mCTT+uJbHOk8sJh3LjA+ofyfQZjw+S6V3d9baSYqL/U79nMWAaed?= =?us-ascii?Q?mgehf2gs6x/F7tXnH3XiaS/5tZZyvdShm3zm6Z62447wxzbSXxLlPSpnMloa?= =?us-ascii?Q?HInx23uyn1TpQIWvFhJnHUYRttEp6gYs2D+UDYrXpuA4JdVfUBDmnMAQKM6i?= =?us-ascii?Q?prhEJPAfoodCjUbasV6YZdPpWvd1wIIG3BuJO4kT5PXBmQWymb4zjBm4xqMK?= =?us-ascii?Q?/w=3D=3D?= X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-Network-Message-Id: ec4de3ea-da04-4169-c4bb-08daf3d21493 X-MS-Exchange-CrossTenant-AuthSource: PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Jan 2023 12:48:04.3068 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: afLCs1N1mfCmAEoVPDR/U4IHKTf1/MHR1WR6LHmoXHqnIEeWB6td+aVHp2xngiYVBHuMotSC1YmXTr28qSB3zcwVS20IXknrkcHWD2iIuNc= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM7PR10MB3176 X-TUID: BQAkJRrNjbcA Am Wed, 11 Jan 2023 09:23:01 +0100 schrieb "Moessbauer, Felix (T CED INW-CN)" : > On Wed, 2023-01-11 at 09:06 +0100, Henning Schild wrote: > > Am Wed, 11 Jan 2023 04:11:32 +0000 > > schrieb Felix Moessbauer : > > > > > When calling python scripts, python automatically creates cache > > > files > > > to speedup future invocations of the same sources. This often > > > happens > > > in postinst scripts, that directly run in the image chroot. The > > > created debian packages do not ship these files, as the debheper > > > scripts remove them before installing. > > > > > > For the rootfs part, we manually have to do it to also not > > > include these in the final image. This patch implements this logic > > > in > > > a custom cleanup postprocess step. As there might be situations > > > where > > > shipping of a subset of the caches is desireable (e.g. readonly > > > rootfs > > > images), we add support to control this logic using > > > ROOTFS_FEATURES. > > > > > > Signed-off-by: Felix Moessbauer > > > --- > > > meta/classes/image.bbclass | 2 +- > > > meta/classes/rootfs.bbclass | 6 ++++++ > > > 2 files changed, 7 insertions(+), 1 deletion(-) > > > > > > diff --git a/meta/classes/image.bbclass > > > b/meta/classes/image.bbclass > > > index 519a2e5..b86a428 100644 > > > --- a/meta/classes/image.bbclass > > > +++ b/meta/classes/image.bbclass > > > @@ -80,7 +80,7 @@ image_do_mounts() { > > > } > > > > > > ROOTFSDIR = "${IMAGE_ROOTFS}" > > > -ROOTFS_FEATURES += "clean-package-cache generate-manifest > > > export-dpkg-status clean-log-files clean-debconf-cache" > > > +ROOTFS_FEATURES += "clean-package-cache clean-pycache > > > generate-manifest export-dpkg-status clean-log-files > > > clean-debconf-cache" ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} > > > ${IMAGE_INSTALL}" ROOTFS_MANIFEST_DEPLOY_DIR ?= > > > "${DEPLOY_DIR_IMAGE}" > > > ROOTFS_DPKGSTATUS_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}" diff --git > > > a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index > > > 786682d..325e7ae 100644 --- a/meta/classes/rootfs.bbclass +++ > > > b/meta/classes/rootfs.bbclass @@ -252,6 +252,12 @@ > > > rootfs_postprocess_clean_debconf_cache() { sudo rm -rf > > > "${ROOTFSDIR}/var/cache/debconf/"* } > > > > > > +ROOTFS_POSTPROCESS_COMMAND += > > > "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-pycache', > > > 'rootfs_postprocess_clean_pycache', '', d)}" > > > +rootfs_postprocess_clean_pycache() { > > > + sudo find ${ROOTFSDIR}/usr -type f -name '*.pyc' > > > -delete -print > > > + sudo find ${ROOTFSDIR}/usr -type d -name '__pycache__' > > > -delete -print +} > > > > Are we sure that this can never be valid content of any package? I > > suggest we double check with dpkg. > > I already checked this. Shipping the __pycache__ folder is a linitan > error [1], shipping any .pyc files is a linitan warning [2]. > > Adding bbwarn here does not make sense either, as we cannot > distinguish between pycache entries from a broken package and ones > created by postinst scripts. Anyways, pyc files are just cache files > and these should not be part of any package or image. Can we not ask dpkg -S for every file before we delete it? Removing files owned by package would likely be wrong. No matter what you might think of the quality of such a package and how many debian rules you cite. We have these kinds of packages, coming from funny vendors and maybe also from weird recipes. I am not worried about packages coming from debian and built with debian tooling. Henning > In case a user really wants to ship .pyc files, he can still disable > this rootfs feature. But the debian ruleset should be our baseline, > not some erroneous behavior that somebody could implement. > > [1] > https://lintian.debian.org/tags/package-installs-python-pycache-dir > [2] > https://lintian.debian.org/tags/source-contains-prebuilt-python-object > > Felix > > > > > Henning > > > > > ROOTFS_POSTPROCESS_COMMAND += > > > "${@bb.utils.contains('ROOTFS_FEATURES', 'generate-manifest', > > > 'rootfs_generate_manifest', '', d)}" rootfs_generate_manifest () { > > > mkdir -p ${ROOTFS_MANIFEST_DEPLOY_DIR} > > >