From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7182122219497062400 X-Received: by 2002:a2e:82c8:0:b0:27b:5631:8a4 with SMTP id n8-20020a2e82c8000000b0027b563108a4mr2082963ljh.326.1672219337698; Wed, 28 Dec 2022 01:22:17 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:ac2:4e8a:0:b0:4c8:8384:83f3 with SMTP id o10-20020ac24e8a000000b004c8838483f3ls2928654lfr.3.-pod-prod-gmail; Wed, 28 Dec 2022 01:22:16 -0800 (PST) X-Google-Smtp-Source: AMrXdXvEGxQjXO/iewtSS0hejcd26QHodJ4GgkEENBsWzg9CfcXoyxNe2ZUWIJDnSRgntfYX4/n3 X-Received: by 2002:a19:f514:0:b0:4b7:2a7:1241 with SMTP id j20-20020a19f514000000b004b702a71241mr6348746lfb.64.1672219336612; Wed, 28 Dec 2022 01:22:16 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672219336; cv=none; d=google.com; s=arc-20160816; b=GI+RLDustnpgsI50JfJnC+BeLogrsZN3UE1uWg4Is8rM8DUvUIGXYDvgEkViQplXvw CjXrL6qoEG7bmUuHkJrCokqmBZD5TvvH5Rh4mZZBzEti9z6v8Kow6cHMWs1apz7J0950 2YNHxCqhjyhdDcUZYVaRAAhlW9KdTzg8uPpzstv2+FmFw8ed9r1BT1NrBTRj0nqFBGp3 pxG+pJlec2ft/NyCFT5208nM7mcc7x21QiGEmgLNcaKVxSP2/32fWVr+GJcwrNJJMkXa x1VBigPyP8OFHbsb5b+sugLMm35Gw9KIxMmZJ/lzuwtmYDKV+3hC97ZdL/7XxnOZtd2k cfpw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:mime-version:content-transfer-encoding:references :in-reply-to:date:cc:to:from:subject:message-id:dkim-signature; bh=ABMdgnlzpXhCK94sLxMRmWtKF2gkGHzULE+XJ8wgWWY=; b=mGjBxFU+UYGhLZoptlI33KUy09nbA/KNe+vtsJPqTh/N+/5jylcNfFJ3bpqTx+XILr sEvnj1I99BXFWn5j1i2VoIIEq4IdQjxc5actYjjNvwvchBExsnIoLS8bJ+jLxVWSgiIR zQOxeS8B+lYQJYTj9V2exw66yPGnoKZKb73HIWGsfAcg6FgR37VWeNgcD5syyNQTgEZd e3LFqmjld4iAHxBiEtEXD2A7p2gzVuDXc+NPFx+f+ddXY5OeOWqCpzfwBW04JTMqPt2Q 8JcC7aF7dUSU9FGfmDSL5kRNq2zDJFXUSHmy/SaefhgZe+1uwlnfXCwfRdefbK4/DMvv M+Pg== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=oQjcavxm; spf=pass (google.com: domain of fm-68982-2022122809221586bea29806419fcf4b-_hhnbr@rts-flowmailer.siemens.com designates 185.136.64.227 as permitted sender) smtp.mailfrom=fm-68982-2022122809221586bea29806419fcf4b-_hHnBR@rts-flowmailer.siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from mta-64-227.siemens.flowmailer.net (mta-64-227.siemens.flowmailer.net. [185.136.64.227]) by gmr-mx.google.com with ESMTPS id g2-20020a056512118200b0048b38f379d7si590565lfr.0.2022.12.28.01.22.16 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 28 Dec 2022 01:22:16 -0800 (PST) Received-SPF: pass (google.com: domain of fm-68982-2022122809221586bea29806419fcf4b-_hhnbr@rts-flowmailer.siemens.com designates 185.136.64.227 as permitted sender) client-ip=185.136.64.227; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=oQjcavxm; spf=pass (google.com: domain of fm-68982-2022122809221586bea29806419fcf4b-_hhnbr@rts-flowmailer.siemens.com designates 185.136.64.227 as permitted sender) smtp.mailfrom=fm-68982-2022122809221586bea29806419fcf4b-_hHnBR@rts-flowmailer.siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: by mta-64-227.siemens.flowmailer.net with ESMTPSA id 2022122809221586bea29806419fcf4b for ; Wed, 28 Dec 2022 10:22:15 +0100 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=fm1; d=siemens.com; i=florian.bezdeka@siemens.com; h=Date:From:Subject:To:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:Cc:References:In-Reply-To; bh=ABMdgnlzpXhCK94sLxMRmWtKF2gkGHzULE+XJ8wgWWY=; b=oQjcavxmftmBkhovoV0vzN2aiRzMu9Qmaw7rKqnMIJvhkjfCb3CRJYjcmI/UVuj6zsYMn6 Vb/miolWhaQVEeBn83b3BMmWfierLcACA1F7+4b+CxSz6OOwm8+MGh8WzolXf2ZZYDCqPdwJ pblXgQh8WKMzSxAap1qaMsFc36zCQ=; Message-ID: <947d479c5619055bc951bf026ff79e28f4887230.camel@siemens.com> Subject: Re: Better way to handle apt cache needed From: Florian Bezdeka To: "Moessbauer, Felix" , "isar-users@googlegroups.com" Cc: "Schild, Henning" Date: Wed, 28 Dec 2022 10:22:15 +0100 In-Reply-To: <371e4d826cca6aaba11a4222fef547b134ed6ce7.camel@siemens.com> References: <371e4d826cca6aaba11a4222fef547b134ed6ce7.camel@siemens.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-68982:519-21489:flowmailer X-TUID: 2af5fpUctS3U Hi all, On Wed, 2022-12-28 at 09:02 +0000, Moessbauer, Felix wrote: > Hi, >=20 > when working with builds that have both many recipes, as well as many > build dependencies, disk usage explodes during the build. As both > preconditions somehow correspond, this results in a quadratic disc > consumption in the number of tasks during the build. >=20 > The root cause for that behavior is the apt cache > (deb_dl_dir_(import|export)), that copies all previously downloaded apt > packages into the WORKDIR of each (bitbake) package. > Given, that a common apt-cache is around 2GB and 8 tasks are run in > parallel, this gives already 16GB for the tasks, and 7 * 2GB for the > buildchroots (host and target), in total ~30GB. >=20 > In one of my projects, we have to work with huge debian packages, > leading to apt-cache sizes around 20GB. As these projects usually also > have to be built on big machines with many cores, you easily get 500GB > of required scratch disk space + a lot of disc accesses for the copy, Thanks for bringing this up. The second aspect ^^ (the IO stuff) is the most annoying aspect in my environment. Using Isar on a distributed file system is slow, near to the point where you refuse to run a build on such a file system. > making it basically impossible to build the project except by limiting > the number of tasks that run in parallel. >=20 > Given that, we should really think about a way to get the disc > consumption back to a linear level. Ideally, we would only use symlinks > or maybe hardlinks to deduplicate. Another option would be to use the > POSIX atomicity guarantees by just renaming packages when inserting > into the cache. I haven't looked into that, so: Are we talking about creation of symlinks instead of "cp" or do we have to do more? Let's try and run some benchmarks/comparisons? >=20 > Anyways, we need a better solution. > Putting Henning as the author of that logic in CC. >=20 > Best regards, > Felix >=20