Re: Better way to handle apt cache needed

["Moessbauer, Felix" <felix.moessbauer@siemens.com>]

On Wed, 2022-12-28 at 13:23 +0300, Uladzimir Bely wrote:
> In mail from среда, 28 декабря 2022 г. 12:45:07 +03 user Moessbauer,
> Felix 
> wrote:
> > On Wed, 2022-12-28 at 10:21 +0100, Baurzhan Ismagulov wrote:
> > > On Wed, Dec 28, 2022 at 09:02:13AM +0000, Moessbauer, Felix
> > > wrote:
> > > > The root cause for that behavior is the apt cache
> > > > (deb_dl_dir_(import|export)), that copies all previously
> > > > downloaded
> > > > apt
> > > > packages into the WORKDIR of each (bitbake) package.
> > > > Given, that a common apt-cache is around 2GB and 8 tasks are
> > > > run in
> > > > parallel, this gives already 16GB for the tasks, and 7 * 2GB
> > > > for
> > > > the
> > > > buildchroots (host and target), in total ~30GB.
> > > 
> > > Thanks Felix for the report. IIRC, it was previously mounted and
> > > was
> > > supposed
> > > to be converted to per-package hardlinks to parallelize sbuild
> > > instances and
> > > ease debugging (by knowing later which exact snapshot was fed to
> > > a
> > > specific
> > > build). We use small (1- / 2-TB) SSDs as job storage and a huge
> > > increase would
> > > have been noticeable... We'll check.
> > 
> > Thanks!
> > 
> > I just noticed, that we even make an additional copy to have the
> > packages inside the sbuild chroot.
> > 
> > This behavior is hard to notice on small and medium sized projects
> > (and
> > given that IOPS are not an issue). But any quadratic behavior will
> > eventually make the build impossible. And as Florian said, many of
> > our
> > ISAR users build in VMs on shared filesystems, where IO is
> > extremely
> > expensive / slow. If we could optimize that, it would be a huge
> > benefit
> > for a lot of users.
> 
> I've just did some measurements:
> 
> - run Isar build for qemuarm64 (8 cores = max 8 build tasks in
> parallel)
> - measured system disk consumption every 5 seconds
> 
> Results:
> - 'downloads/deb' directory finally takse 480MiB
> - After the build finished, disk space decreased by ~9GiB
> - During the build maximum disk space decrease was ~16GiB)
> 
> It means, that we really had about 16 - 9 = 7GiB of space temporarly
> used by 
> parallel builds and it really corresponds to 8 * 2 * 480MiB.
> 
> So, the main goal now is to minimize this value.

Even the 7GiB should be further reduced, as some major contributor to
that is the apt-cache for the buildchroots and the images itself, which
is never cleaned up. While this is not so much an issue for CI systems,
it is definitely annoying on local builds. Especially, when working on
multiple ISAR based projects.

I would recommend to have a look at the build/tmp folder using ncdu -x
(-x to not cross filesystems / temporary mounts).

Felix

> 
> > 
> > Felix
> > 
> > > With kind regards,
> > > Baurzhan
> 
> 
> 
>