Re: Better way to handle apt cache needed

["Moessbauer, Felix" <felix.moessbauer@siemens.com>]

On Wed, 2022-12-28 at 10:21 +0100, Baurzhan Ismagulov wrote:
> On Wed, Dec 28, 2022 at 09:02:13AM +0000, Moessbauer, Felix wrote:
> > The root cause for that behavior is the apt cache
> > (deb_dl_dir_(import|export)), that copies all previously downloaded
> > apt
> > packages into the WORKDIR of each (bitbake) package.
> > Given, that a common apt-cache is around 2GB and 8 tasks are run in
> > parallel, this gives already 16GB for the tasks, and 7 * 2GB for
> > the
> > buildchroots (host and target), in total ~30GB.
> 
> Thanks Felix for the report. IIRC, it was previously mounted and was
> supposed
> to be converted to per-package hardlinks to parallelize sbuild
> instances and
> ease debugging (by knowing later which exact snapshot was fed to a
> specific
> build). We use small (1- / 2-TB) SSDs as job storage and a huge
> increase would
> have been noticeable... We'll check.

Thanks!

I just noticed, that we even make an additional copy to have the
packages inside the sbuild chroot.

This behavior is hard to notice on small and medium sized projects (and
given that IOPS are not an issue). But any quadratic behavior will
eventually make the build impossible. And as Florian said, many of our
ISAR users build in VMs on shared filesystems, where IO is extremely
expensive / slow. If we could optimize that, it would be a huge benefit
for a lot of users.

Felix

> 
> With kind regards,
> Baurzhan