From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7182122219497062400 X-Received: by 2002:a05:6512:224d:b0:4b5:84a1:1b6c with SMTP id i13-20020a056512224d00b004b584a11b6cmr2267701lfu.560.1672676168663; Mon, 02 Jan 2023 08:16:08 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:ac2:4e8a:0:b0:4c8:8384:83f3 with SMTP id o10-20020ac24e8a000000b004c8838483f3ls83533lfr.3.-pod-prod-gmail; Mon, 02 Jan 2023 08:16:07 -0800 (PST) X-Google-Smtp-Source: AMrXdXt/ZrzaOrCAnLsEuBWioSv4KgAd0Pemk57xWtbiEPiB0tzkhY+eD6AQJUGAvnhvGIAWk0e2 X-Received: by 2002:ac2:4478:0:b0:4a4:68b7:dedb with SMTP id y24-20020ac24478000000b004a468b7dedbmr14065650lfl.55.1672676167577; Mon, 02 Jan 2023 08:16:07 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1672676167; cv=pass; d=google.com; s=arc-20160816; b=HxTNGXFhgkW3MYLOMP808CSmyVgBFgoHugoq1w444H0dPg4eEPLfQbIulf32N53aFj vElzGs8ItxWdaYBo9uWqODV1phAOtymIPcN9VP8r4A4F66MEKrUqDQanECx6U6nevBfn hjCq8Rbdlv2ZLdCcvwNcgNvAdrqAFD60zbQxoC+bRJ7jcHmZ0bJcwXRvV+M/eJxSsw8N YWVM66jH1IQuch0eadKowVepYf6UjWmKchexvKTAXTQH57gWD+9Jt1XgwMyjnvWHZA3n 9GhF/5nUdrwdtPOYF3PtvMAg13PDnm0ib3zCkZYaZQh85uNe05PkinZFo0cXsmZvt0Xq EHeg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:content-transfer-encoding:references:in-reply-to :message-id:subject:cc:to:from:date:dkim-signature; bh=UEeXqKYzzh56uiaLJk1lx+wVa4Vt6mwIw5Zn4sRL0Sg=; b=EFdS/nDmCQuiem2bHz6z28WWskk2KShW6Sg6h3ElNplrSTmh9ysJ4LBUXxJ6KfLuV7 F0ZI0DHJl3lGYTcDp2iUoS0MDHMIxtTayJErVIADP6wrg6FwJyVO28NujYVgum41Dqn+ qYHlLOLn1T9p/XbR5x+8egHbibSwRZjb4WQ7JU2bIuEeIU1iCDQv+tyPtVPTclTjlVid HcRxJUsbfiRSwVrSLu/Wb+zVKGjlxSedgDj67U6bfqcNBYD/piuw+MRftvgJAkqMEWF3 9Myxs0PmCkABBQ8eRxAwMVFEHs7d2SvnCYy+4AdbIjfdKY/BKoPnGGlmkkkQTZvx5s12 BJdQ== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=BtTpHgJT; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of henning.schild@siemens.com designates 40.107.15.89 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from EUR01-DB5-obe.outbound.protection.outlook.com (mail-db5eur01on2089.outbound.protection.outlook.com. [40.107.15.89]) by gmr-mx.google.com with ESMTPS id w2-20020a05651234c200b004b49cc7bf6asi948496lfr.9.2023.01.02.08.16.07 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 02 Jan 2023 08:16:07 -0800 (PST) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 40.107.15.89 as permitted sender) client-ip=40.107.15.89; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=BtTpHgJT; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of henning.schild@siemens.com designates 40.107.15.89 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mS+DLvGbOG9VEzieacW3Vx3G4+kckKbyMSRQk3CPNq42EYjioGuUbIZWeXqn4Pv9+AAQ9P7hKlEwBoBV1lOj8qLH7h4lyUcs2sywAjs637JvRFfc0OBj+oRXj+h+DErsZD0RwLyLj+WL8Si3wQ3hTyFj91P663e8n4gJTmS1fVVQXX45/D0IFRh5I5P2dOvVoD8u7UZ3CxLyBdSeyG7TnbFVPlGdseskXa6OwzlZF9lS2Acm4hPPk6HIuQI/bfppLHnMdtaAzEIY9ViBIKNNvGqwPTz2qBgrGLJi7QCTVFEmwFnSg6+NEV/74mHMpwUqS01Ehus5uUwyftVYphlSLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=UEeXqKYzzh56uiaLJk1lx+wVa4Vt6mwIw5Zn4sRL0Sg=; b=SV4RgJAwSU7ZtzQO2XUKbRqlVZOXZ7IEMitC9f8AUCETakD9fmED945OGnODq3cTll0ZRxr65oGa4lt4yMwKyuV7YlCTt67oeqrzQGBcNgYslDF81gcx94L4iQhAQ9quSA0dMsav7j3sR/OcLIHTc5XavfA9qNPcS/NZDcCLeibFZcrdrXsxtJBOuN/hVOUUoJZ3m0Orb140PwuZFXGxdU68Yy0UoSqJ4angAffMjHsYqXA5ogC53z9lNNP4Qs2UtZBw1KbDMxv6IzvhIbx5a5Ax8ZxM5qiVH5yT8Kd09lj4bwc/wlJ1enE96yszE+xbJfM4NHMmiZ8LbnkbZbX7wA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=siemens.com; dmarc=pass action=none header.from=siemens.com; dkim=pass header.d=siemens.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=UEeXqKYzzh56uiaLJk1lx+wVa4Vt6mwIw5Zn4sRL0Sg=; b=BtTpHgJTuP8rb6ArhWAEe0t8t6zB9ky0dseclM2m4NndQT06LxGl8/hHiabR58lJZdXJkJq8PqhSH28o6gnPVhUK1ncSRrWhm7dlCcmg5u9Xk+5LuOg8E56aIiUCibB/BkylXUcjLu/gpFNgROKHo2vVQJ8xQS/tQAN+TD7n5ypugjJvdn2lKu1TwydXcT0tVoC6Ut21QtjstR3+g1+fBp+t+MXvu3Ulj1UFtgwNiJnsViBmiSi68st6/rmVM9gzTRzExftzeIx0HbXdtUIt62HJh3Si8R0H1NX80M4CGpV72D1NVHkBCVURnHHQjDtsrGbtxU5hIK+oHYlUG3iAOw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=siemens.com; Received: from PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:102:269::8) by GV1PR10MB6610.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:150:82::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5944.19; Mon, 2 Jan 2023 16:16:05 +0000 Received: from PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM ([fe80::bdf0:fdeb:f955:bc79]) by PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM ([fe80::bdf0:fdeb:f955:bc79%3]) with mapi id 15.20.5944.019; Mon, 2 Jan 2023 16:16:05 +0000 Date: Mon, 2 Jan 2023 17:15:57 +0100 From: Henning Schild To: "Moessbauer, Felix (T CED INW-CN)" Cc: "isar-users@googlegroups.com" Subject: Re: Better way to handle apt cache needed Message-ID: <20230102171557.42d08791@md1za8fc.ad001.siemens.net> In-Reply-To: <371e4d826cca6aaba11a4222fef547b134ed6ce7.camel@siemens.com> References: <371e4d826cca6aaba11a4222fef547b134ed6ce7.camel@siemens.com> X-Mailer: Claws Mail 4.1.0 (GTK 3.24.35; x86_64-pc-linux-gnu) Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-ClientProxiedBy: CH0PR04CA0033.namprd04.prod.outlook.com (2603:10b6:610:77::8) To PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:102:269::8) Return-Path: henning.schild@siemens.com MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PA4PR10MB5780:EE_|GV1PR10MB6610:EE_ X-MS-Office365-Filtering-Correlation-Id: d0993770-500b-4309-7932-08daecdca66d X-LD-Processed: 38ae3bcd-9579-4fd4-adda-b42e1495d55a,ExtAddr X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 8m08hZ9AQQuosQahjZd7szEAaVSKjRv6D26Y4w/HSJ8tibm22KuZsnJOyj/r1HhxHt207iN7vY53VislBTDaqAyYKB60PtBe0Y73K/Hsyp4ZjDefMQcjv2bIogUMW7rUdsk/kikBEEz5yH13sjHp4POCINbX+9bj3AxIaP9l1RiLqlO95f93vzDee0tUPYeiK9G81aUwT7utRMQ9NfXy4MNtR4pquDdj6shrySXe42OqvZfWxTGl7T13t7hI3q2SkbPKJOH+C3lqco3bLU10OyatYUV10cXqJwKV3gWrDx8Ivf3XObi093pfo7olDj0lv5QT2SBHQXgAUPtmhl3K1C0dbUSte7yxAoGE4clfToiPjKIBF9z7yyLwND35Rb0hk4/4yJxgnkYY4hE43XkE/vxvCfwh/miN9LkVLfK7MEINrOqOJTS8NsBzoi6IpMIqi7Pp0P8adXd1gFrypOd9CnInLSWxZr4JkFTrofKspFC89cqM2z5QD2bNWqKXB0Zo/j4KL+FHfBwUttJcjhw3zHNvNh5YNCtx0gCoXQzIrhm8g0N5CjU6c3ws/OFrI/bNeipy5YTVwtWyXGkdlDS+CIzTlFVfZeH+yUjlA8O5Lc2Z3fL1E8fUUU1xqvQwwR9AX8r0NHR3OfSEIGT5+5vp1I6Z9bz4ezVMHAl+Fo37Eb3nYJhvpHTOrT8lARdRb6GIgV6PyJFKPmADe8W1G/yuAg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230022)(4636009)(396003)(376002)(136003)(346002)(366004)(39860400002)(451199015)(5660300002)(2906002)(44832011)(8936002)(8676002)(4326008)(41300700001)(6862004)(66899015)(478600001)(6636002)(66476007)(316002)(66946007)(66556008)(6486002)(1076003)(26005)(9686003)(6512007)(6506007)(6666004)(83380400001)(82960400001)(38100700002)(186003)(86362001)(22166006)(43043002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?9jxRj8Rzo44TOF1RWIZxzAlPo+6tI9c4MhFV5jtfbqHZH/mOPOFSVd5E7BJi?= =?us-ascii?Q?RheMYvkd71KSAP6ejx1WQT4m8HSg2v528G4An0TdRjaCKqZEdnMZODGsu7Ov?= =?us-ascii?Q?50rxUqz9ue0YMGkNtsVvr1yCDRj0oW9COCQZ6r0EXr9JdLyiT1MJiBL6wU3F?= =?us-ascii?Q?CBywzYJIgWxshbnFT8s+gfnqOmftVDVkr0BVDfU6NlZRby/wKzdF7kSu9SyO?= =?us-ascii?Q?Jknn/XWe7SUPbI3bP4CvziChVAkClpB8Rw5RNOVhCvG6uHwQu/3d1BHj0mlh?= =?us-ascii?Q?erIGmJNQZnMlewbnTVMVmlb3utVjryuLa8+Q38+xU4pw7/Kz1HKcLN/y1+nP?= =?us-ascii?Q?A4DvGz+cjqe6ydm8mWtkGSK7GdGKBDIfYdwqidu1YK5lYL+YJE8cwIgL5f4Y?= =?us-ascii?Q?GuRul46EzinBXTRqwSF1Dcj2YCqYleg2TxRpMCiSch6e5EOEhcTdWZR4O2YT?= =?us-ascii?Q?e3glmvUY1zTZ8YH9IrYoMUqxvJwU2w72772BfCIIfbiBRZJPfXGmolETeMaq?= =?us-ascii?Q?fxbGSNK2t40hGEI9ouSq/JY8bMYyD330Lbm5FZkzXBiZ8RoG+nEIEojTurxF?= =?us-ascii?Q?ziPVuqbyEpczzfc86FaeDtOyhYsoqwZhMlJAG9Spay9BTAe4R/X/BAHRntPJ?= =?us-ascii?Q?p3GPrrgfNHVweTDQp8zhjUc8CcYjiTyTRUpxOc0y342MSaW/MLyhv/Ny+Dtv?= =?us-ascii?Q?f0W4aoaPP9r9ixpvBoAk0vlG22ASDPaXgVQIQrZscc0sfpBvkkyeA/syjyey?= =?us-ascii?Q?twmKpE0/vzkHu0vXFvPGqA4DPNJQXJjTjeNVCP1GG9/3T5AFbSRg30okUlDT?= =?us-ascii?Q?ktjMLJ3MqKJQ8ZHnSqhD/wOAnejk5QNlVi2WqbV6H5aQ1X/5d/ttUFwNyVzX?= =?us-ascii?Q?6zLIxFZItlLiuyTY03bjfwEWps1MSfYh9hPrpHJ0dMZIVzFbDxKcUms9u3G/?= =?us-ascii?Q?OCoAmMNkrMcMbfgxH8wWMeZ3uAnrhC0qRN6tU6xwX37+Jioym2FlyHH474cn?= =?us-ascii?Q?On29p1dVAoG0J5rBkEy1cDBZmbdP6l/9cCSI9w8PTErMTYG/lxyT4QAfcaVM?= =?us-ascii?Q?AQeVv0AJpTZY+gjvJHHUjgIxFs6y34shpq9GJwApMl+Uu20kUxl1HCe7HpqF?= =?us-ascii?Q?rvv4JvCPQYa+eBNznOLg175a6IsxVRPX+q8GwT9gJdLlsg+xTsee3XblG1AT?= =?us-ascii?Q?K1sCR4OF+PIqaCBgNM4o9df3Fh5tpq+wpfALuq8BGbTYWypaj6cz9ZRGGyK7?= =?us-ascii?Q?lbZAtc+q0arwexnzkmzvNH0DlKLGp1bpr25MJ5OvesXiNNprs543joMK15uN?= =?us-ascii?Q?Js8ML1BfI+Ce9abo6mg9cjxKrfG757VtgQbep7cNjvIgZuoqcMRHGfmkfYX4?= =?us-ascii?Q?VSe+86aYnEvuQ0Bunnpxzi1s9aqeQTjc2bPMqXqS3A92wxAYfgpau3eQ849H?= =?us-ascii?Q?46t1ac3BbPmj7XaSzYe1u+JmmNnsR5b/S8pEU5u2Wyikq1M3Xl4L1NYjia30?= =?us-ascii?Q?zc8E6fLRNnCQfGC8+ZdIB8d1jfoZwnDRkRfmlEF84QUYy9IGy6JDkm5vhY+O?= =?us-ascii?Q?GTbphb4A0Ys0l39IDcb+kYFGu+2Fl8BwpiWICwgkFTIhdSWHbT5vFFF6S4A5?= =?us-ascii?Q?Pg=3D=3D?= X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-Network-Message-Id: d0993770-500b-4309-7932-08daecdca66d X-MS-Exchange-CrossTenant-AuthSource: PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Jan 2023 16:16:05.7217 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: fxJBUTTA/IIEG1wajPfuP9FWg6rG42q1xUQpYwWY4Sgsdm1tEdCpT3bCnwH6FOv6MdoWLC7CXAaWV1kufbDA//z8eaHt6wRYnb7tCeP/Xc8= X-MS-Exchange-Transport-CrossTenantHeadersStamped: GV1PR10MB6610 X-TUID: 08zL9bNtT0lr Am Wed, 28 Dec 2022 10:02:13 +0100 schrieb "Moessbauer, Felix (T CED INW-CN)" : > Hi, > > when working with builds that have both many recipes, as well as many > build dependencies, disk usage explodes during the build. As both > preconditions somehow correspond, this results in a quadratic disc > consumption in the number of tasks during the build. > > The root cause for that behavior is the apt cache > (deb_dl_dir_(import|export)), that copies all previously downloaded > apt packages into the WORKDIR of each (bitbake) package. > Given, that a common apt-cache is around 2GB and 8 tasks are run in > parallel, this gives already 16GB for the tasks, and 7 * 2GB for the > buildchroots (host and target), in total ~30GB. > > In one of my projects, we have to work with huge debian packages, > leading to apt-cache sizes around 20GB. As these projects usually also > have to be built on big machines with many cores, you easily get 500GB > of required scratch disk space + a lot of disc accesses for the copy, > making it basically impossible to build the project except by limiting > the number of tasks that run in parallel. > > Given that, we should really think about a way to get the disc > consumption back to a linear level. Ideally, we would only use > symlinks or maybe hardlinks to deduplicate. Another option would be > to use the POSIX atomicity guarantees by just renaming packages when > inserting into the cache. > > Anyways, we need a better solution. > Putting Henning as the author of that logic in CC. Thanks for bringing that up. I did not yet realize that problem but it sounds very real and valid. That thread here is already very long just looking at it and not having read much of it. All i can say in my defense is that sbuild is likely what took as from linear to multiplication. So technically i have nothing to do with any of this ;). Henning > Best regards, > Felix