From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7182796543154454528 X-Received: by 2002:a05:6512:2528:b0:4b6:e956:54a2 with SMTP id be40-20020a056512252800b004b6e95654a2mr3020746lfb.663.1673020717886; Fri, 06 Jan 2023 07:58:37 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6512:1182:b0:4b5:3cdf:5a65 with SMTP id g2-20020a056512118200b004b53cdf5a65ls446868lfr.2.-pod-prod-gmail; Fri, 06 Jan 2023 07:58:36 -0800 (PST) X-Google-Smtp-Source: AMrXdXtW0u+hGhszqng7TqbeFrj6eRNv39C+ZE0FO1Fc0RbY8xhMDDbH12mTSrVQ+h3GpjCLQ9BG X-Received: by 2002:a05:6512:3c88:b0:4b4:e4a1:2fc6 with SMTP id h8-20020a0565123c8800b004b4e4a12fc6mr19132814lfv.68.1673020716530; Fri, 06 Jan 2023 07:58:36 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1673020716; cv=pass; d=google.com; s=arc-20160816; b=AFthLmFX7VFZT6UwY1qbCJAz9PasD/8lWTKWYDxjPTqXiUlg831DnD54uoVJxLbhcz eZmtgKCPHnUBTAYC8xFLv7daGFZlgjWQOrygYAH1C3dyWrvz6L2DJG7Ppl1uClICktA0 0Pfk1YqAGJYOaUjKwB33l2sj6g2QKK4Do1HM5zwPAxKM31nbq+jS9TY6dnhzG0tlZV9e zvpxmOk1PwHpgdrYIRsg2v0Tbw1vLngRdbF1XDvBCPZ22nafLqurPhr3J48HmgOiKa6S LDH8LF58WXXANy15HSIzXulUMFdIh0illv8taY0ypIXPottugiVkAomMxv3tZ2zqOEN4 wdFg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:content-transfer-encoding:references:in-reply-to :message-id:subject:cc:to:from:date:dkim-signature; bh=JOLLuYGYcMGAcW6SHIz/VDEsSUCYhrSuE7RnOv+BudY=; b=qpnOBBYPj7wFQTQBBtb1uZRb4JavLq0MlPrHR58drJcc21kuJkrwUgFJBCWjc08rpp OGIu3h5YJcpNDtqt3AFwhmd8Jv6a/cLxld/msv/qotEKTGs5LvyHzOzperAGs7m84Uhn V6j19Kdg5GfWLDKLdjAbVRCzH8F1N8lFE0i8AIx2yN2CUxzhTxHmiAGUWsn/ey/KL50a 15pEKQLy0HMwsn/CVjXCjmhSPtACI5lu8GiXOQ86lihDC649RbBzFVjnUE71erJXdT9W ya456bt1+bT78s7GZxJ0Cuo20znicWJRqGcB7JON6O0QdSZPY2SkjoYkBihIg1T/PsyF H79g== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=y3wmDGsH; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of henning.schild@siemens.com designates 40.107.249.72 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from EUR02-DB5-obe.outbound.protection.outlook.com (mail-db5eur02on2072.outbound.protection.outlook.com. [40.107.249.72]) by gmr-mx.google.com with ESMTPS id s4-20020a056512202400b004abdb5d1128si49144lfs.2.2023.01.06.07.58.36 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 06 Jan 2023 07:58:36 -0800 (PST) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 40.107.249.72 as permitted sender) client-ip=40.107.249.72; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=y3wmDGsH; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of henning.schild@siemens.com designates 40.107.249.72 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PYCcE+fy8ihjTM1IMyE2Re8n5wRIM/Lsm8LN15xTo4L4dynGMr5GJQ5xCN1GjMxZqYGIJu3UDqlW0kDCwN1ZO4B0Quq5kpNEFA7dpSZIC+t7NaHHHlXRGwh4/+zPtAQYdG5xu4eLzG2YLYor0eY6FcPdqNTHvNHNFqZfvSxf26DeTt0g3+Zj2wWbmefFAvSaOh/8T8PD35I5zmlMIR4XroHG13xWUfLlgIc0TnP8jqB14FHKRVEpiI4qPZpxRtY0YiNYki2ayKfsUBwQfHD/ODRS0NUYOkAckyWsDglFpiHwY3fZxFJ8Ee+56Nf3ZevQSNTymdJ7g+n4QTlH4OXmSQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=JOLLuYGYcMGAcW6SHIz/VDEsSUCYhrSuE7RnOv+BudY=; b=izEJzBuIYyEVbMn9vaJChZN4j3CqwXNug5xQIE0U722c+RgEdR62eldchbb6gzIDEN/jihuUDPdB8azwYVKFV6bJAxjzyFGeovGFQu2e59dCVGJCN/EDjiT+H3ug3jL4bO6gi8NDwW5J6ke/FRlN33tnIx4CM4CLt+ebV7oBSy1u+eHrL/n7+3E2XBQoxd1MA4Ll1C4Cp4gjnEnmTjsIaf+PYUS1kd+uPusvkwo32T3LMWV+l6Wm4zl3iuv2nm3bvS4fmVnJ466S9W3YxyqDmtTwN3bIsVg8ai4ntYj+kVnZ8pFEmUhwSnRjCeBLy9OAEK3QwFRQW1MySWTaW+X1jg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=siemens.com; dmarc=pass action=none header.from=siemens.com; dkim=pass header.d=siemens.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JOLLuYGYcMGAcW6SHIz/VDEsSUCYhrSuE7RnOv+BudY=; b=y3wmDGsHffaFpWZ+8h4PFQ1mN3XBOaNnr9TITupQiNe7qpysC0EgM6i9sos6nVmeFtBEFcvecVupWR3AhtxB+DVVECv+Iio+i/htSUWIfSPUAno20wovD2nN6fuV3wbE7hF+pDo0t52Ubf1fTgqeC4OaZ9He0ukv5V3GlVC5FjMMYs1IuvsORYa5Xq9rtlqY3W2jidUHyOVlqAx4IXS252TyOOzRYtHCsppXcFYpIpqaoG1OOvbNyLkAUniPzAdNRvaDl9On36mxw7aAYwD3x1xz8uKRXWsThDRC34akJQvpUwatBAoQDwKwm+0gtZ7UH3BZ0e16wRTD438Ox72Bpg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=siemens.com; Received: from PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:102:269::8) by DU0PR10MB5335.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:10:34e::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5944.19; Fri, 6 Jan 2023 15:58:35 +0000 Received: from PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM ([fe80::bdf0:fdeb:f955:bc79]) by PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM ([fe80::bdf0:fdeb:f955:bc79%3]) with mapi id 15.20.5944.019; Fri, 6 Jan 2023 15:58:34 +0000 Date: Fri, 6 Jan 2023 16:58:26 +0100 From: Henning Schild To: Uladzimir Bely Cc: isar-users@googlegroups.com Subject: Re: [PATCH v2 3/3] Changes for a faster build using less disk space Message-ID: <20230106165826.1346da94@md1za8fc.ad001.siemens.net> In-Reply-To: <20230106064809.10412-4-ubely@ilbers.de> References: <20230106064809.10412-1-ubely@ilbers.de> <20230106064809.10412-4-ubely@ilbers.de> X-Mailer: Claws Mail 4.1.0 (GTK 3.24.35; x86_64-pc-linux-gnu) Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-ClientProxiedBy: CH0P220CA0030.NAMP220.PROD.OUTLOOK.COM (2603:10b6:610:ef::28) To PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:102:269::8) Return-Path: henning.schild@siemens.com MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PA4PR10MB5780:EE_|DU0PR10MB5335:EE_ X-MS-Office365-Filtering-Correlation-Id: 703c84b6-de19-4885-6673-08daeffedd9e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: wzd+POR1htKVFe1x7EGi3+lKa9MYPt/3H3J0wjymsWVs77Pb5j48XY8k9roAJ+5/raFa/CzN6sJaBUm+Lj72MuVcDwe+aTUB1SSIvuW/X4qvbX4BX8qvbamkXzqfzj+qH2LOhq0B2q0cGQwZ18B6JUQbnMgTXUD05H4Hk/03B5G2krJzzk1YKvxvEfWE/gQfIQoghkocc/A/h4VC9TPKatTgeukiHtv9xvQYQlPFiXK0Yek/9V9XvCq10MOu+Uxlb2P6BfDSfE4bc26spTv7GQ+3zXLobVhLm15dYR/KHkUstZigDSfRrJRIEeEWgnI0K2PJdvvkTJBBQhaq3C+B0BDt2O6lIBQFO8nKMKBsuvl0SlCeph1DklJ+3xS5NYkuu+dY4ljTuF4BaAvImauxA/e6tkmkxu7y5hGnsEmhWs1VeFu00y+twKn9OxOpCW5Bsi1k8woqPGP3wwOJJG86/PN1RsnIJxaCRC9LxiW02LGbMnyYzdV/Y5PAU1hRx+yZvZA4oW6ZGvFpDri6RsgTwFbqz4nqcxgAyfGXAEEIiGFEnxsMwhKtRn7qlTGGO8lM4BS6h+mcRwVB1ZmwtayjWCIeNQ5gtc7BFsqaAwpIKzK6njnaqxdF3ohFDMqXuD632t6+2xIPT9ZCFOrplVZnIA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230022)(4636009)(396003)(346002)(366004)(136003)(39860400002)(376002)(451199015)(5660300002)(2906002)(44832011)(8936002)(41300700001)(316002)(478600001)(4326008)(8676002)(66556008)(66476007)(6916009)(66946007)(6486002)(9686003)(6512007)(6666004)(6506007)(83380400001)(1076003)(186003)(38100700002)(82960400001)(86362001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?/7Lr0TOjSxx9XHOHcnFgFxsFiBqmcfOvGpk14WoCewbPxMmZzuNeeccgqXXP?= =?us-ascii?Q?k4t5AAdheIwcyi6GUqVG+E9aOV7r9KpPDcJ+SVbYTHOk3OMgQU7V7Xli4fWH?= =?us-ascii?Q?BmfZ/soTv1zb1al6koNt6H2/bcvCQbtnpSmweDizNN5jE5gzK3yQQn4Jz5e6?= =?us-ascii?Q?BSZRqwt4TXnRN9YOWPggIOilS3wHVJVstiKxb+7StGaoVHBYYTnj+bOGZ14m?= =?us-ascii?Q?HDYixH8e19UIV7IhNjphV9J4+2QFzz3SBT57V/DQHlwmJDQwyoKIcfAMVPvQ?= =?us-ascii?Q?B+SJgjvRWBrdpjPo8EAMjf6pLqx4+xh+v7iw7DfUWf8vrdrSygdcmygZJ77Y?= =?us-ascii?Q?2aRF9fWZhQcBdbIj92YujCeZDy0XcsifoHh4DWNr1vHpebOVzb6gAELi+zoq?= =?us-ascii?Q?XLAiIuh0tnawIWd/3xRibhDj54DgCs7/wFdjK8RJkjcPEKBt5oyoYckZ3HDw?= =?us-ascii?Q?s8XyfuJAr0PgZpTJGNE5Pw5ms2e8DB51MW+nTxIuIs1X9DSlsPPkXFliNVHl?= =?us-ascii?Q?4lZTM6rOSZQPF4SL3x2SZsxfcaJp3t8/lhMKlOVerT3dkqbeq+w19azosE9j?= =?us-ascii?Q?at+I+iyIUYA3J7ZOH90Y3pDrrZf7DgL1cIVXbOZ13Z91PM4F8KWNby3pOqnM?= =?us-ascii?Q?ZJiuzhGEjRdvoyzznnKdsJXS/k2VETGk6iAH6JAdxm46c4DqcVPfNlQoJ4e1?= =?us-ascii?Q?MqM08yxYgeJ8EZiiyd4Al/Mj2L+v8N+m8JR9dR9UTK/Y8uJjOfapIqMybRg/?= =?us-ascii?Q?7e76SdFL2WXc650KxQqZbtn5GZuJW51sE2nDTG8UYySUdKk9nrsVXSJQYeQ6?= =?us-ascii?Q?d00+Q2o3taGmtdurtQT6eaDd8GHYzgj6vlRoy/ONHDBJq0l/l0qmd9dh2CAX?= =?us-ascii?Q?tUh54kvtJ3lmYU/1P7GzYjEcWCGXWTfI3bk3RuuUWjxjQpkKR/TJno9DvVUc?= =?us-ascii?Q?FzYT2XSu2oJ6hppo2OPWSoGK4eO4Z24490DdAoEi0WlLlCshJCBSkoMLsvsw?= =?us-ascii?Q?vIRVGTHoBbLjX68j7/uEj4axBXXPaaUKxmIT/3E5WDC9bieqKZpcgoGRYiWi?= =?us-ascii?Q?CoLDUSbrDgxy0RNbTV5c5CTN49ZIKqHJ1TxGJUbqwl/A/EDd67i9+5HNKnvx?= =?us-ascii?Q?1y/Stkc3igavw0LLTTN+IA/OQR1u35jYyvdO/PER5rZgm1WO/jfLi9nyQEec?= =?us-ascii?Q?2NYDlPuMnIN1K/GIi8clVbMu8sHHWQEU4/mNuj0ctazpWInXHizsUndqrOg+?= =?us-ascii?Q?eNf72ErBgcOitQXEuh88J8d8CGe2m/9Cb9kdlMUsI2M4Xh7ZGffLkiqyCGwI?= =?us-ascii?Q?RCsQfV1KBoEPJKVaoTmcnTFhaWTfiM/vZICWyLSXX7Eut+kcPnh3HgHIO+YI?= =?us-ascii?Q?Vru52vIvwL7AIpcqcG7uL3wmWJjP990sSzyHA09Kp/E+yYM/pK0cxR8x3cIT?= =?us-ascii?Q?Te5+TJuyse+GgPN5jEXdk1rKzR8Xi6netdGaxOXIlBqJM4PPnqZNchkk1q5M?= =?us-ascii?Q?KmFGhjCrQvgV8io/8D0j0DHLK5lGR3vjwlGsNBm7cGinlewZsp8QfLbnB7sn?= =?us-ascii?Q?BmyTFZm1rLEkwxpv9+uyIOuGgwp+84OO8RLa9uo1RHWWiuPjRTRUVL6Mk1Y0?= =?us-ascii?Q?xJ77A1Hnqb3ODIbrURoSS1qpZRaPTMEmdKb+DotSseQ13rEqNZZpD3Wey5yN?= =?us-ascii?Q?BVbClA=3D=3D?= X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-Network-Message-Id: 703c84b6-de19-4885-6673-08daeffedd9e X-MS-Exchange-CrossTenant-AuthSource: PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jan 2023 15:58:34.8660 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: WuRJcNZ7LQz+gjH2htcXq2Fuj5inY1vn2SYep5/Pr9pzK9NEJF14bZm2jx6UQZ/eI4Bg+IH3QozcLMj4RdFxnrYr8lzEgIp2E3Ovg8o+/6s= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR10MB5335 X-TUID: wXN4t7bTf9m9 Am Fri, 6 Jan 2023 07:48:09 +0100 schrieb Uladzimir Bely : > From: "Roberto A. Foglietta" > > Faster build for larger projects with a lot of debian packages > > This patch leverages few ways to accomplish the task to optimise > the large buildings. > > Put a CACHEDIR.TAG file in each cache folder in such a way every > tar that uses the option --exclude-caches could be avoided > to include debian packages saving a lot of time and disk space. > > Signed-off-by: Roberto A. Foglietta > Signed-off-by: Uladzimir Bely > --- > meta/classes/deb-dl-dir.bbclass | 2 +- > meta/classes/rootfs.bbclass | 3 ++- > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 7 ++++++- > 3 files changed, 9 insertions(+), 3 deletions(-) > > diff --git a/meta/classes/deb-dl-dir.bbclass > b/meta/classes/deb-dl-dir.bbclass index 7db25251..3afad4e8 100644 > --- a/meta/classes/deb-dl-dir.bbclass > +++ b/meta/classes/deb-dl-dir.bbclass > @@ -102,7 +102,7 @@ deb_dl_dir_export() { > -maxdepth 1 -type f -iname '*\.deb' |\ > while read p; do > # skip files from a previous export > - [ -f "${pc}/${p##*/}" ] && continue > + [ -e "${pc}/${p##*/}" ] && continue > # can not reuse bitbake function here, this is basically > # "repo_contains_package" > package=$(find "${REPO_ISAR_DIR}"/"${DISTRO}" -name > ${p##*/}) diff --git a/meta/classes/rootfs.bbclass > b/meta/classes/rootfs.bbclass index 786682d9..f274443d 100644 > --- a/meta/classes/rootfs.bbclass > +++ b/meta/classes/rootfs.bbclass > @@ -308,7 +308,8 @@ rootfs_install_sstate_prepare() { > # so we use some mount magic to prevent that > mkdir -p ${WORKDIR}/mnt/rootfs > sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro > - sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar --one-file-system > rootfs > + lopts="--one-file-system --exclude-caches" > + sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar $lopts rootfs > sudo umount ${WORKDIR}/mnt/rootfs > sudo chown $(id -u):$(id -g) rootfs.tar > } > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index > cb0079ec..44019f95 100644 --- > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -395,6 > +395,10 @@ EOSUDO > # Cleanup apt cache > sudo -Es chroot "${ROOTFSDIR}" /usr/bin/apt-get -y clean > + > + # Don't include apt cache into sstate cache by marking the > directory > + # with a CACHEDIR.TAG file with a special signature > + sudo -Es sh -c "echo Signature: 8a477f597d28d172789f06886806bc55 What is that "special signature" all about? Does anyone later care to remove the file again, or do we risk it ending up in the final rootfs? I find this a little weird and would prefer if the calls to tar would simply exclude this by path and not by magic tool specific file. It could be made a central function so we have all those tar calls in one place. In this patch i see three places ... hard to maintain. Henning > > ${ROOTFSDIR}/var/cache/apt/archives/CACHEDIR.TAG" } > > addtask bootstrap before do_build after do_generate_keyrings > @@ -405,7 +409,8 @@ SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize" > > bootstrap_sstate_prepare() { > # this runs in SSTATE_BUILDDIR, which will be deleted > automatically > - sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar > --one-file-system $(basename "${ROOTFSDIR}") > + lopts="--one-file-system --exclude-caches" > + sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar $lopts > $(basename "${ROOTFSDIR}") sudo chown $(id -u):$(id -g) bootstrap.tar > } >