Re: Better way to handle apt cache needed

[Uladzimir Bely <ubely@ilbers.de>]

In the email from Wednesday, 28 December 2022 12:02:13 +03 user Moessbauer, 
Felix wrote:
> Hi,
> 
> when working with builds that have both many recipes, as well as many
> build dependencies, disk usage explodes during the build. As both
> preconditions somehow correspond, this results in a quadratic disc
> consumption in the number of tasks during the build.
> 
> The root cause for that behavior is the apt cache
> (deb_dl_dir_(import|export)), that copies all previously downloaded apt
> packages into the WORKDIR of each (bitbake) package.
> Given, that a common apt-cache is around 2GB and 8 tasks are run in
> parallel, this gives already 16GB for the tasks, and 7 * 2GB for the
> buildchroots (host and target), in total ~30GB.
> 
> In one of my projects, we have to work with huge debian packages,
> leading to apt-cache sizes around 20GB. As these projects usually also
> have to be built on big machines with many cores, you easily get 500GB
> of required scratch disk space + a lot of disc accesses for the copy,
> making it basically impossible to build the project except by limiting
> the number of tasks that run in parallel.
> 
> Given that, we should really think about a way to get the disc
> consumption back to a linear level. Ideally, we would only use symlinks
> or maybe hardlinks to deduplicate. Another option would be to use the
> POSIX atomicity guarantees by just renaming packages when inserting
> into the cache.
> 
> Anyways, we need a better solution.
> Putting Henning as the author of that logic in CC.
> 
> Best regards,
> Felix

Hi all

I'd like just to mention unfinished patchset that I was working on earlier. It 
was last sent to maillist as `[PATCH v3 0/5] Improving base-apt usage PoC`.

The idea was to predownload all possible package dependencies to 'base-apt' 
repo first and use it (as "file:///path/to/base-apt") in sources list.

In this case (e.g. "file://..." source), as far as I remember, apt doesn't 
need packages to be "downloaded" to /var/cache/apt and just use them directly.

Since that time, of course, Isar changed (for example, now we have host/target 
splitted base-apt), so the patchset need to be updated. Also it requires some 
cleanup and improvements (technically, we don't need $DL_DIR/deb at all, since 
we use local base-apt repo).

Anyway, such kind of local-repo-based approach would be a good solution for 
high disk usage problem.

-- 
Uladzimir Bely