Re: Better way to handle apt cache needed

[Henning Schild <henning.schild@siemens.com>]

Am Wed, 28 Dec 2022 10:02:13 +0100
schrieb "Moessbauer, Felix (T CED INW-CN)"
<felix.moessbauer@siemens.com>:

> Hi,
> 
> when working with builds that have both many recipes, as well as many
> build dependencies, disk usage explodes during the build. As both
> preconditions somehow correspond, this results in a quadratic disc
> consumption in the number of tasks during the build.
> 
> The root cause for that behavior is the apt cache
> (deb_dl_dir_(import|export)), that copies all previously downloaded
> apt packages into the WORKDIR of each (bitbake) package.
> Given, that a common apt-cache is around 2GB and 8 tasks are run in
> parallel, this gives already 16GB for the tasks, and 7 * 2GB for the
> buildchroots (host and target), in total ~30GB.
>
> In one of my projects, we have to work with huge debian packages,
> leading to apt-cache sizes around 20GB. As these projects usually also
> have to be built on big machines with many cores, you easily get 500GB
> of required scratch disk space + a lot of disc accesses for the copy,
> making it basically impossible to build the project except by limiting
> the number of tasks that run in parallel.
> 
> Given that, we should really think about a way to get the disc
> consumption back to a linear level. Ideally, we would only use
> symlinks or maybe hardlinks to deduplicate. Another option would be
> to use the POSIX atomicity guarantees by just renaming packages when
> inserting into the cache.
> 
> Anyways, we need a better solution.
> Putting Henning as the author of that logic in CC.

Thanks for bringing that up. I did not yet realize that problem but it
sounds very real and valid.

That thread here is already very long just looking at it and not having
read much of it.

All i can say in my defense is that sbuild is likely what took as from
linear to multiplication. So technically i have nothing to do with any
of this ;).

Henning

> Best regards,
> Felix