Re: Better way to handle apt cache needed

[Florian Bezdeka <florian.bezdeka@siemens.com>]

Hi all,

On Wed, 2022-12-28 at 09:02 +0000, Moessbauer, Felix wrote:
> Hi,
> 
> when working with builds that have both many recipes, as well as many
> build dependencies, disk usage explodes during the build. As both
> preconditions somehow correspond, this results in a quadratic disc
> consumption in the number of tasks during the build.
> 
> The root cause for that behavior is the apt cache
> (deb_dl_dir_(import|export)), that copies all previously downloaded apt
> packages into the WORKDIR of each (bitbake) package.
> Given, that a common apt-cache is around 2GB and 8 tasks are run in
> parallel, this gives already 16GB for the tasks, and 7 * 2GB for the
> buildchroots (host and target), in total ~30GB.
> 
> In one of my projects, we have to work with huge debian packages,
> leading to apt-cache sizes around 20GB. As these projects usually also
> have to be built on big machines with many cores, you easily get 500GB
> of required scratch disk space + a lot of disc accesses for the copy,

Thanks for bringing this up. The second aspect ^^ (the IO stuff) is the
most annoying aspect in my environment. Using Isar on a distributed
file system is slow, near to the point where you refuse to run a build
on such a file system.

> making it basically impossible to build the project except by limiting
> the number of tasks that run in parallel.
> 
> Given that, we should really think about a way to get the disc
> consumption back to a linear level. Ideally, we would only use symlinks
> or maybe hardlinks to deduplicate. Another option would be to use the
> POSIX atomicity guarantees by just renaming packages when inserting
> into the cache.

I haven't looked into that, so: Are we talking about creation of
symlinks instead of "cp" or do we have to do more? Let's try and run
some benchmarks/comparisons?

> 
> Anyways, we need a better solution.
> Putting Henning as the author of that logic in CC.
> 
> Best regards,
> Felix
>