Better way to handle apt cache needed

Better way to handle apt cache needed

[Moessbauer, Felix]

Hi,

when working with builds that have both many recipes, as well as many
build dependencies, disk usage explodes during the build. As both
preconditions somehow correspond, this results in a quadratic disc
consumption in the number of tasks during the build.

The root cause for that behavior is the apt cache
(deb_dl_dir_(import|export)), that copies all previously downloaded apt
packages into the WORKDIR of each (bitbake) package.
Given, that a common apt-cache is around 2GB and 8 tasks are run in
parallel, this gives already 16GB for the tasks, and 7 * 2GB for the
buildchroots (host and target), in total ~30GB.

In one of my projects, we have to work with huge debian packages,
leading to apt-cache sizes around 20GB. As these projects usually also
have to be built on big machines with many cores, you easily get 500GB
of required scratch disk space + a lot of disc accesses for the copy,
making it basically impossible to build the project except by limiting
the number of tasks that run in parallel.

Given that, we should really think about a way to get the disc
consumption back to a linear level. Ideally, we would only use symlinks
or maybe hardlinks to deduplicate. Another option would be to use the
POSIX atomicity guarantees by just renaming packages when inserting
into the cache.

Anyways, we need a better solution.
Putting Henning as the author of that logic in CC.

Best regards,
Felix

Re: Better way to handle apt cache needed

[Baurzhan Ismagulov]

On Wed, Dec 28, 2022 at 09:02:13AM +0000, Moessbauer, Felix wrote:
> The root cause for that behavior is the apt cache
> (deb_dl_dir_(import|export)), that copies all previously downloaded apt
> packages into the WORKDIR of each (bitbake) package.
> Given, that a common apt-cache is around 2GB and 8 tasks are run in
> parallel, this gives already 16GB for the tasks, and 7 * 2GB for the
> buildchroots (host and target), in total ~30GB.

Thanks Felix for the report. IIRC, it was previously mounted and was supposed
to be converted to per-package hardlinks to parallelize sbuild instances and
ease debugging (by knowing later which exact snapshot was fed to a specific
build). We use small (1- / 2-TB) SSDs as job storage and a huge increase would
have been noticeable... We'll check.

With kind regards,
Baurzhan

Re: Better way to handle apt cache needed

[Florian Bezdeka]

Hi all,

On Wed, 2022-12-28 at 09:02 +0000, Moessbauer, Felix wrote:
> Hi,
> 
> when working with builds that have both many recipes, as well as many
> build dependencies, disk usage explodes during the build. As both
> preconditions somehow correspond, this results in a quadratic disc
> consumption in the number of tasks during the build.
> 
> The root cause for that behavior is the apt cache
> (deb_dl_dir_(import|export)), that copies all previously downloaded apt
> packages into the WORKDIR of each (bitbake) package.
> Given, that a common apt-cache is around 2GB and 8 tasks are run in
> parallel, this gives already 16GB for the tasks, and 7 * 2GB for the
> buildchroots (host and target), in total ~30GB.
> 
> In one of my projects, we have to work with huge debian packages,
> leading to apt-cache sizes around 20GB. As these projects usually also
> have to be built on big machines with many cores, you easily get 500GB
> of required scratch disk space + a lot of disc accesses for the copy,

Thanks for bringing this up. The second aspect ^^ (the IO stuff) is the
most annoying aspect in my environment. Using Isar on a distributed
file system is slow, near to the point where you refuse to run a build
on such a file system.

> making it basically impossible to build the project except by limiting
> the number of tasks that run in parallel.
> 
> Given that, we should really think about a way to get the disc
> consumption back to a linear level. Ideally, we would only use symlinks
> or maybe hardlinks to deduplicate. Another option would be to use the
> POSIX atomicity guarantees by just renaming packages when inserting
> into the cache.

I haven't looked into that, so: Are we talking about creation of
symlinks instead of "cp" or do we have to do more? Let's try and run
some benchmarks/comparisons?

> 
> Anyways, we need a better solution.
> Putting Henning as the author of that logic in CC.
> 
> Best regards,
> Felix
> 

Re: Better way to handle apt cache needed

[Moessbauer, Felix]

On Wed, 2022-12-28 at 10:21 +0100, Baurzhan Ismagulov wrote:
> On Wed, Dec 28, 2022 at 09:02:13AM +0000, Moessbauer, Felix wrote:
> > The root cause for that behavior is the apt cache
> > (deb_dl_dir_(import|export)), that copies all previously downloaded
> > apt
> > packages into the WORKDIR of each (bitbake) package.
> > Given, that a common apt-cache is around 2GB and 8 tasks are run in
> > parallel, this gives already 16GB for the tasks, and 7 * 2GB for
> > the
> > buildchroots (host and target), in total ~30GB.
> 
> Thanks Felix for the report. IIRC, it was previously mounted and was
> supposed
> to be converted to per-package hardlinks to parallelize sbuild
> instances and
> ease debugging (by knowing later which exact snapshot was fed to a
> specific
> build). We use small (1- / 2-TB) SSDs as job storage and a huge
> increase would
> have been noticeable... We'll check.

Thanks!

I just noticed, that we even make an additional copy to have the
packages inside the sbuild chroot.

This behavior is hard to notice on small and medium sized projects (and
given that IOPS are not an issue). But any quadratic behavior will
eventually make the build impossible. And as Florian said, many of our
ISAR users build in VMs on shared filesystems, where IO is extremely
expensive / slow. If we could optimize that, it would be a huge benefit
for a lot of users.

Felix

> 
> With kind regards,
> Baurzhan

Re: Better way to handle apt cache needed

[Uladzimir Bely]

In mail from среда, 28 декабря 2022 г. 12:45:07 +03 user Moessbauer, Felix 
wrote:
> On Wed, 2022-12-28 at 10:21 +0100, Baurzhan Ismagulov wrote:
> > On Wed, Dec 28, 2022 at 09:02:13AM +0000, Moessbauer, Felix wrote:
> > > The root cause for that behavior is the apt cache
> > > (deb_dl_dir_(import|export)), that copies all previously downloaded
> > > apt
> > > packages into the WORKDIR of each (bitbake) package.
> > > Given, that a common apt-cache is around 2GB and 8 tasks are run in
> > > parallel, this gives already 16GB for the tasks, and 7 * 2GB for
> > > the
> > > buildchroots (host and target), in total ~30GB.
> > 
> > Thanks Felix for the report. IIRC, it was previously mounted and was
> > supposed
> > to be converted to per-package hardlinks to parallelize sbuild
> > instances and
> > ease debugging (by knowing later which exact snapshot was fed to a
> > specific
> > build). We use small (1- / 2-TB) SSDs as job storage and a huge
> > increase would
> > have been noticeable... We'll check.
> 
> Thanks!
> 
> I just noticed, that we even make an additional copy to have the
> packages inside the sbuild chroot.
> 
> This behavior is hard to notice on small and medium sized projects (and
> given that IOPS are not an issue). But any quadratic behavior will
> eventually make the build impossible. And as Florian said, many of our
> ISAR users build in VMs on shared filesystems, where IO is extremely
> expensive / slow. If we could optimize that, it would be a huge benefit
> for a lot of users.

I've just did some measurements:

- run Isar build for qemuarm64 (8 cores = max 8 build tasks in parallel)
- measured system disk consumption every 5 seconds

Results:
- 'downloads/deb' directory finally takse 480MiB
- After the build finished, disk space decreased by ~9GiB
- During the build maximum disk space decrease was ~16GiB)

It means, that we really had about 16 - 9 = 7GiB of space temporarly used by 
parallel builds and it really corresponds to 8 * 2 * 480MiB.

So, the main goal now is to minimize this value.

> 
> Felix
> 
> > With kind regards,
> > Baurzhan

Re: Better way to handle apt cache needed

[Moessbauer, Felix]

On Wed, 2022-12-28 at 13:23 +0300, Uladzimir Bely wrote:
> In mail from среда, 28 декабря 2022 г. 12:45:07 +03 user Moessbauer,
> Felix 
> wrote:
> > On Wed, 2022-12-28 at 10:21 +0100, Baurzhan Ismagulov wrote:
> > > On Wed, Dec 28, 2022 at 09:02:13AM +0000, Moessbauer, Felix
> > > wrote:
> > > > The root cause for that behavior is the apt cache
> > > > (deb_dl_dir_(import|export)), that copies all previously
> > > > downloaded
> > > > apt
> > > > packages into the WORKDIR of each (bitbake) package.
> > > > Given, that a common apt-cache is around 2GB and 8 tasks are
> > > > run in
> > > > parallel, this gives already 16GB for the tasks, and 7 * 2GB
> > > > for
> > > > the
> > > > buildchroots (host and target), in total ~30GB.
> > > 
> > > Thanks Felix for the report. IIRC, it was previously mounted and
> > > was
> > > supposed
> > > to be converted to per-package hardlinks to parallelize sbuild
> > > instances and
> > > ease debugging (by knowing later which exact snapshot was fed to
> > > a
> > > specific
> > > build). We use small (1- / 2-TB) SSDs as job storage and a huge
> > > increase would
> > > have been noticeable... We'll check.
> > 
> > Thanks!
> > 
> > I just noticed, that we even make an additional copy to have the
> > packages inside the sbuild chroot.
> > 
> > This behavior is hard to notice on small and medium sized projects
> > (and
> > given that IOPS are not an issue). But any quadratic behavior will
> > eventually make the build impossible. And as Florian said, many of
> > our
> > ISAR users build in VMs on shared filesystems, where IO is
> > extremely
> > expensive / slow. If we could optimize that, it would be a huge
> > benefit
> > for a lot of users.
> 
> I've just did some measurements:
> 
> - run Isar build for qemuarm64 (8 cores = max 8 build tasks in
> parallel)
> - measured system disk consumption every 5 seconds
> 
> Results:
> - 'downloads/deb' directory finally takse 480MiB
> - After the build finished, disk space decreased by ~9GiB
> - During the build maximum disk space decrease was ~16GiB)
> 
> It means, that we really had about 16 - 9 = 7GiB of space temporarly
> used by 
> parallel builds and it really corresponds to 8 * 2 * 480MiB.
> 
> So, the main goal now is to minimize this value.

Even the 7GiB should be further reduced, as some major contributor to
that is the apt-cache for the buildchroots and the images itself, which
is never cleaned up. While this is not so much an issue for CI systems,
it is definitely annoying on local builds. Especially, when working on
multiple ISAR based projects.

I would recommend to have a look at the build/tmp folder using ncdu -x
(-x to not cross filesystems / temporary mounts).

Felix

> 
> > 
> > Felix
> > 
> > > With kind regards,
> > > Baurzhan
> 
> 
> 
> 

Re: Better way to handle apt cache needed

[Roberto A. Foglietta]

On Wed, 28 Dec 2022 at 12:04, Moessbauer, Felix
<felix.moessbauer@siemens.com> wrote:
>
> On Wed, 2022-12-28 at 13:23 +0300, Uladzimir Bely wrote:
> > In mail from среда, 28 декабря 2022 г. 12:45:07 +03 user Moessbauer,
> > Felix
> > wrote:
> > > On Wed, 2022-12-28 at 10:21 +0100, Baurzhan Ismagulov wrote:
> > > > On Wed, Dec 28, 2022 at 09:02:13AM +0000, Moessbauer, Felix
> > > > wrote:
> > > > > The root cause for that behavior is the apt cache
> > > > > (deb_dl_dir_(import|export)), that copies all previously
> > > > > downloaded
> > > > > apt
> > > > > packages into the WORKDIR of each (bitbake) package.
> > > > > Given, that a common apt-cache is around 2GB and 8 tasks are
> > > > > run in
> > > > > parallel, this gives already 16GB for the tasks, and 7 * 2GB
> > > > > for
> > > > > the
> > > > > buildchroots (host and target), in total ~30GB.
> > > >
> > > > Thanks Felix for the report. IIRC, it was previously mounted and
> > > > was
> > > > supposed
> > > > to be converted to per-package hardlinks to parallelize sbuild
> > > > instances and
> > > > ease debugging (by knowing later which exact snapshot was fed to
> > > > a
> > > > specific
> > > > build). We use small (1- / 2-TB) SSDs as job storage and a huge
> > > > increase would
> > > > have been noticeable... We'll check.
> > >
> > > Thanks!
> > >
> > > I just noticed, that we even make an additional copy to have the
> > > packages inside the sbuild chroot.
> > >
> > > This behavior is hard to notice on small and medium sized projects
> > > (and
> > > given that IOPS are not an issue). But any quadratic behavior will
> > > eventually make the build impossible. And as Florian said, many of
> > > our
> > > ISAR users build in VMs on shared filesystems, where IO is
> > > extremely
> > > expensive / slow. If we could optimize that, it would be a huge
> > > benefit
> > > for a lot of users.
> >
> > I've just did some measurements:
> >
> > - run Isar build for qemuarm64 (8 cores = max 8 build tasks in
> > parallel)
> > - measured system disk consumption every 5 seconds
> >
> > Results:
> > - 'downloads/deb' directory finally takse 480MiB
> > - After the build finished, disk space decreased by ~9GiB
> > - During the build maximum disk space decrease was ~16GiB)
> >
> > It means, that we really had about 16 - 9 = 7GiB of space temporarly
> > used by
> > parallel builds and it really corresponds to 8 * 2 * 480MiB.
> >
> > So, the main goal now is to minimize this value.
>
> Even the 7GiB should be further reduced, as some major contributor to
> that is the apt-cache for the buildchroots and the images itself, which
> is never cleaned up. While this is not so much an issue for CI systems,
> it is definitely annoying on local builds. Especially, when working on
> multiple ISAR based projects.
>
> I would recommend to have a look at the build/tmp folder using ncdu -x
> (-x to not cross filesystems / temporary mounts).


Hi all,

I did some changes to reduce disk usage and to speed up the building.
The results are quite impressing so, before everything else I am going
to show you the numbers:

results before and after the changes

43954 Mb (max)   |  8657 Mb (max)
26548 Mb (rest)  |  4118 Mb (rest)
 3741 Mb (deb)   |  3741 Mb (deb)
  820 Mb (wic)   |   820 Mb (wic)
11789 Mb (cache) |   579 Mb (cache)
time: 8m33s      | time: 4m34s

The changes has been committed on the default branch (evo) on my ISAR fork

 https://github.com/robang74/isar

* 9fd5282 - changes for a faster build using less disk space, p.1
* 1b6ac1e - bugfix: no sstate archive obtainable, will run full task instead
* 2cc1854 - bugfix: do_copy_boot_files was able to fail but then set -e

The two bug fixes are necessary to test for regression because the top
commit changes the sstate cache creation. The top commit uses hard
physical links to obtain these results, so it may not work in some
cases like distributed or networked filesystems which do not support
hard links (however, they might be smart enough to do a bare copy but
I cannot grant that).

However, this is the reason because I titled part one the patch. I am
thinking of addressing the issue in a more radical way but for the
moment this seems quite interesting.
I wish to have some feedback about the limitation about using uncommon
filesystems and which one have a nice fallback to address unsupported
features.

Looking forward your opinions, R-

Re: Better way to handle apt cache needed

[Uladzimir Bely]

In mail from пятница, 30 декабря 2022 г. 02:15:33 +03 user Roberto A. 
Foglietta wrote:
> On Wed, 28 Dec 2022 at 12:04, Moessbauer, Felix
> 
> <felix.moessbauer@siemens.com> wrote:
> > On Wed, 2022-12-28 at 13:23 +0300, Uladzimir Bely wrote:
> > > In mail from среда, 28 декабря 2022 г. 12:45:07 +03 user Moessbauer,
> > > Felix
> > > 
> > > wrote:
> > > > On Wed, 2022-12-28 at 10:21 +0100, Baurzhan Ismagulov wrote:
> > > > > On Wed, Dec 28, 2022 at 09:02:13AM +0000, Moessbauer, Felix
> > > > > 
> > > > > wrote:
> > > > > > The root cause for that behavior is the apt cache
> > > > > > (deb_dl_dir_(import|export)), that copies all previously
> > > > > > downloaded
> > > > > > apt
> > > > > > packages into the WORKDIR of each (bitbake) package.
> > > > > > Given, that a common apt-cache is around 2GB and 8 tasks are
> > > > > > run in
> > > > > > parallel, this gives already 16GB for the tasks, and 7 * 2GB
> > > > > > for
> > > > > > the
> > > > > > buildchroots (host and target), in total ~30GB.
> > > > > 
> > > > > Thanks Felix for the report. IIRC, it was previously mounted and
> > > > > was
> > > > > supposed
> > > > > to be converted to per-package hardlinks to parallelize sbuild
> > > > > instances and
> > > > > ease debugging (by knowing later which exact snapshot was fed to
> > > > > a
> > > > > specific
> > > > > build). We use small (1- / 2-TB) SSDs as job storage and a huge
> > > > > increase would
> > > > > have been noticeable... We'll check.
> > > > 
> > > > Thanks!
> > > > 
> > > > I just noticed, that we even make an additional copy to have the
> > > > packages inside the sbuild chroot.
> > > > 
> > > > This behavior is hard to notice on small and medium sized projects
> > > > (and
> > > > given that IOPS are not an issue). But any quadratic behavior will
> > > > eventually make the build impossible. And as Florian said, many of
> > > > our
> > > > ISAR users build in VMs on shared filesystems, where IO is
> > > > extremely
> > > > expensive / slow. If we could optimize that, it would be a huge
> > > > benefit
> > > > for a lot of users.
> > > 
> > > I've just did some measurements:
> > > 
> > > - run Isar build for qemuarm64 (8 cores = max 8 build tasks in
> > > parallel)
> > > - measured system disk consumption every 5 seconds
> > > 
> > > Results:
> > > - 'downloads/deb' directory finally takse 480MiB
> > > - After the build finished, disk space decreased by ~9GiB
> > > - During the build maximum disk space decrease was ~16GiB)
> > > 
> > > It means, that we really had about 16 - 9 = 7GiB of space temporarly
> > > used by
> > > parallel builds and it really corresponds to 8 * 2 * 480MiB.
> > > 
> > > So, the main goal now is to minimize this value.
> > 
> > Even the 7GiB should be further reduced, as some major contributor to
> > that is the apt-cache for the buildchroots and the images itself, which
> > is never cleaned up. While this is not so much an issue for CI systems,
> > it is definitely annoying on local builds. Especially, when working on
> > multiple ISAR based projects.
> > 
> > I would recommend to have a look at the build/tmp folder using ncdu -x
> > (-x to not cross filesystems / temporary mounts).
> 
> Hi all,
> 
> I did some changes to reduce disk usage and to speed up the building.
> The results are quite impressing so, before everything else I am going
> to show you the numbers:
> 
> results before and after the changes
> 
> 43954 Mb (max)   |  8657 Mb (max)
> 26548 Mb (rest)  |  4118 Mb (rest)
>  3741 Mb (deb)   |  3741 Mb (deb)
>   820 Mb (wic)   |   820 Mb (wic)
> 11789 Mb (cache) |   579 Mb (cache)
> time: 8m33s      | time: 4m34s
> 
> The changes has been committed on the default branch (evo) on my ISAR fork
> 
>  https://github.com/robang74/isar
> 
> * 9fd5282 - changes for a faster build using less disk space, p.1
> * 1b6ac1e - bugfix: no sstate archive obtainable, will run full task instead
> * 2cc1854 - bugfix: do_copy_boot_files was able to fail but then set -e
> 
> The two bug fixes are necessary to test for regression because the top
> commit changes the sstate cache creation. The top commit uses hard
> physical links to obtain these results, so it may not work in some
> cases like distributed or networked filesystems which do not support
> hard links (however, they might be smart enough to do a bare copy but
> I cannot grant that).
> 
> However, this is the reason because I titled part one the patch. I am
> thinking of addressing the issue in a more radical way but for the
> moment this seems quite interesting.
> I wish to have some feedback about the limitation about using uncommon
> filesystems and which one have a nice fallback to address unsupported
> features.
> 
> Looking forward your opinions, R-

Hello Roberto.

I also did some similar improvements. They are not so radical as yours and the 
benefit is not so big. But anyway, I'm sending 1st version of patchset. For 
the 2nd version I plan to play with your idea with CACHEDIR.TAG and `tar -
exclude-caches` option and to improve work with additional internal sbuild 
copying that you've already implemented.

Re: Better way to handle apt cache needed

[Moessbauer, Felix]

On Fri, 2022-12-30 at 00:15 +0100, Roberto A. Foglietta wrote:
> Hi all,
> 
> I did some changes to reduce disk usage and to speed up the building.
> The results are quite impressing so, before everything else I am
> going
> to show you the numbers:
> 
> results before and after the changes
> 
> 43954 Mb (max)   |  8657 Mb (max)
> 26548 Mb (rest)  |  4118 Mb (rest)
>  3741 Mb (deb)   |  3741 Mb (deb)
>   820 Mb (wic)   |   820 Mb (wic)
> 11789 Mb (cache) |   579 Mb (cache)
> time: 8m33s      | time: 4m34s
> 
> The changes has been committed on the default branch (evo) on my ISAR
> fork
> 
>  https://github.com/robang74/isar
> 
> * 9fd5282 - changes for a faster build using less disk space, p.1

Hi Roberto,

I tested your patches. In general, the implemented pattern looks good
and also result in significantly shorter build times on virtual file
systems.

But I discovered some issues on partial rebuilds. Example:
| ln: will not overwrite just-created '/build/tmp/deploy/buildchroot-
target/debian-bullseye-amd64/var/cache/apt/archives/libnet-ssleay-
perl_1.88-3+b1_arm64.deb' with '/build/downloads/deb/debian-
bullseye/libnet-ssleay-perl_1.88-3+b1_arm64.deb'

This happens in the do_install_npm in the npm class of meta-iot2050
[1].
It could also be an issue with an older sstate cache artifact, or
related to the specific implementation in the NPM class.
Needs further investigation.

[1]
https://github.com/siemens/meta-iot2050/blob/master/classes/npm.bbclass#L105

Best regards,
Felix

> * 1b6ac1e - bugfix: no sstate archive obtainable, will run full task
> instead
> * 2cc1854 - bugfix: do_copy_boot_files was able to fail but then set
> -e
> 
> The two bug fixes are necessary to test for regression because the
> top
> commit changes the sstate cache creation. The top commit uses hard
> physical links to obtain these results, so it may not work in some
> cases like distributed or networked filesystems which do not support
> hard links (however, they might be smart enough to do a bare copy but
> I cannot grant that).
> 
> However, this is the reason because I titled part one the patch. I am
> thinking of addressing the issue in a more radical way but for the
> moment this seems quite interesting.
> I wish to have some feedback about the limitation about using
> uncommon
> filesystems and which one have a nice fallback to address unsupported
> features.
> 
> Looking forward your opinions, R-

Re: Better way to handle apt cache needed

[Roberto A. Foglietta]

On Fri, 30 Dec 2022 at 05:38, Uladzimir Bely <ubely@ilbers.de> wrote:

>
> Hello Roberto.
>
> I also did some similar improvements. They are not so radical as yours and the
> benefit is not so big. But anyway, I'm sending 1st version of patchset. For
> the 2nd version I plan to play with your idea with CACHEDIR.TAG and `tar -
> exclude-caches` option and to improve work with additional internal sbuild
> copying that you've already implemented.
>

About the use of hard links, I do not think that it is a problem
because in build we can find 4 rootfs folders in which there are
everything supposed to be on a Linux rootfs included links of every
nature

4114 build/tmp/work/debian-bullseye-amd64/isar-bootstrap-target/1.0-r0/rootfs
4501 build/tmp/work/debian-bullseye-amd64/sbuild-chroot-target/1.0-r0/rootfs
4559 build/tmp/work/debian-bullseye-amd64/buildchroot-target/1.0-r0/rootfs
9064 build/tmp/work/debian-bullseye-amd64/eval-image-debx86/1.0-r0/rootfs

This makes me think that trying to use ISAR on every filesystem that
does not support all this stuff would not be able to create a suitable
rootfs. So, the use of ln -P should not be a problem.

Best regards, R-

Re: Better way to handle apt cache needed

[Roberto A. Foglietta]

On Fri, 30 Dec 2022 at 07:05, Moessbauer, Felix
<felix.moessbauer@siemens.com> wrote:
>
> On Fri, 2022-12-30 at 00:15 +0100, Roberto A. Foglietta wrote:
> > Hi all,
> >
> > I did some changes to reduce disk usage and to speed up the building.
> > The results are quite impressing so, before everything else I am
> > going
> > to show you the numbers:
> >
> > results before and after the changes
> >
> > 43954 Mb (max)   |  8657 Mb (max)
> > 26548 Mb (rest)  |  4118 Mb (rest)
> >  3741 Mb (deb)   |  3741 Mb (deb)
> >   820 Mb (wic)   |   820 Mb (wic)
> > 11789 Mb (cache) |   579 Mb (cache)
> > time: 8m33s      | time: 4m34s
> >
> > The changes has been committed on the default branch (evo) on my ISAR
> > fork
> >
> >  https://github.com/robang74/isar
> >
> > * 9fd5282 - changes for a faster build using less disk space, p.1
>
> Hi Roberto,
>
> I tested your patches. In general, the implemented pattern looks good
> and also result in significantly shorter build times on virtual file
> systems.
>
> But I discovered some issues on partial rebuilds. Example:
> | ln: will not overwrite just-created '/build/tmp/deploy/buildchroot-
> target/debian-bullseye-amd64/var/cache/apt/archives/libnet-ssleay-
> perl_1.88-3+b1_arm64.deb' with '/build/downloads/deb/debian-
> bullseye/libnet-ssleay-perl_1.88-3+b1_arm64.deb'
>
> This happens in the do_install_npm in the npm class of meta-iot2050
> [1].
> It could also be an issue with an older sstate cache artifact, or
> related to the specific implementation in the NPM class.
> Needs further investigation.
>
> [1]
> https://github.com/siemens/meta-iot2050/blob/master/classes/npm.bbclass#L105

Hi Felix, try als with the 2nd part which do not force overwriting of
exporting packages

281bcdd - changes for a faster build using less disk space, p.2

I got this error in building the iot2050 project as proposed in README.md:

./kas-container build kas-iot2050-example.yml:kas/opt/sdk.yml

ERROR: python-absl-0.15.0-2-r0 do_fetch: Fetcher failure: Recipe uses
a floating tag/branch without a fixed SRCREV yet doesn't call
bb.fetch2.get_srcrev() (use SRCPV in PV for OE).
ERROR: Logfile of failure stored in:
/build/tmp/work/iot2050-debian-arm64/python-absl/0.15.0-2-r0/temp/log.do_fetch.746
ERROR: Task (/build/../work/meta-coral/recipes-python/python-absl/python-absl_0.15.0-2.bb:do_fetch)
failed with exit code '1'

The only change that I make, is to use my ISAR and obviously I
converted the syntax to the new bitbake 2.0

Best regards, R-

Re: Better way to handle apt cache needed

[Moessbauer, Felix]

On Fri, 2022-12-30 at 09:27 +0100, Roberto A. Foglietta wrote:
> On Fri, 30 Dec 2022 at 07:05, Moessbauer, Felix
> <felix.moessbauer@siemens.com> wrote:
> > 
> > On Fri, 2022-12-30 at 00:15 +0100, Roberto A. Foglietta wrote:
> > > Hi all,
> > > 
> > > I did some changes to reduce disk usage and to speed up the
> > > building.
> > > The results are quite impressing so, before everything else I am
> > > going
> > > to show you the numbers:
> > > 
> > > results before and after the changes
> > > 
> > > 43954 Mb (max)   |  8657 Mb (max)
> > > 26548 Mb (rest)  |  4118 Mb (rest)
> > >  3741 Mb (deb)   |  3741 Mb (deb)
> > >   820 Mb (wic)   |   820 Mb (wic)
> > > 11789 Mb (cache) |   579 Mb (cache)
> > > time: 8m33s      | time: 4m34s
> > > 
> > > The changes has been committed on the default branch (evo) on my
> > > ISAR
> > > fork
> > > 
> > >  https://github.com/robang74/isar
> > > 
> > > * 9fd5282 - changes for a faster build using less disk space, p.1
> > 
> > Hi Roberto,
> > 
> > I tested your patches. In general, the implemented pattern looks
> > good
> > and also result in significantly shorter build times on virtual
> > file
> > systems.
> > 
> > But I discovered some issues on partial rebuilds. Example:
> > > ln: will not overwrite just-created
> > > '/build/tmp/deploy/buildchroot-
> > target/debian-bullseye-amd64/var/cache/apt/archives/libnet-ssleay-
> > perl_1.88-3+b1_arm64.deb' with '/build/downloads/deb/debian-
> > bullseye/libnet-ssleay-perl_1.88-3+b1_arm64.deb'
> > 
> > This happens in the do_install_npm in the npm class of meta-iot2050
> > [1].
> > It could also be an issue with an older sstate cache artifact, or
> > related to the specific implementation in the NPM class.
> > Needs further investigation.
> > 
> > [1]
> > https://github.com/siemens/meta-iot2050/blob/master/classes/npm.bbclass#L105
> 
> Hi Felix, try als with the 2nd part which do not force overwriting of
> exporting packages
> 
> 281bcdd - changes for a faster build using less disk space, p.2

It looks like this did the trick. At least during the last hour I did
not see issues anymore.

> 
> I got this error in building the iot2050 project as proposed in
> README.md:
> 
> ./kas-container build kas-iot2050-example.yml:kas/opt/sdk.yml
> 
> ERROR: python-absl-0.15.0-2-r0 do_fetch: Fetcher failure: Recipe uses
> a floating tag/branch without a fixed SRCREV yet doesn't call
> bb.fetch2.get_srcrev() (use SRCPV in PV for OE).

This is expected, as the meta-coral layer is NOT prepared for bitbake
2.0 yet (same as meta-iot2050). Here, the reason is that bitbake 2.0
does no longer allow git tags as SRCREV, but only SHAs.

> ERROR: Logfile of failure stored in:
> /build/tmp/work/iot2050-debian-arm64/python-absl/0.15.0-2-
> r0/temp/log.do_fetch.746
> ERROR: Task (/build/../work/meta-coral/recipes-python/python-
> absl/python-absl_0.15.0-2.bb:do_fetch)
> failed with exit code '1'
> 
> The only change that I make, is to use my ISAR and obviously I
> converted the syntax to the new bitbake 2.0

Please don't mix things up as this makes the review and testing of
patches much harder. Bitbake 2.0 is not yet integrated in ISAR next, so
please also base your patches on top on ISAR next only.

And please also try to send the patches to the ML (at least the final
versions). While Github is good to get a first overview, it is
impossible to review there (at least in the current working mode) and
this is also not how contributions shall be done in ISAR.

Felix

> 
> Best regards, R-

Re: Better way to handle apt cache needed

[Roberto A. Foglietta]

On Fri, 30 Dec 2022 at 00:15, Roberto A. Foglietta
<roberto.foglietta@gmail.com> wrote:

> Hi all,
>
> I did some changes to reduce disk usage and to speed up the building.
> The results are quite impressing so, before everything else I am going
> to show you the numbers:
>
> results before and after the changes
>
> 43954 Mb (max)   |  8657 Mb (max)
> 26548 Mb (rest)  |  4118 Mb (rest)
>  3741 Mb (deb)   |  3741 Mb (deb)
>   820 Mb (wic)   |   820 Mb (wic)
> 11789 Mb (cache) |   579 Mb (cache)
> time: 8m33s      | time: 4m34s
>
> The changes has been committed on the default branch (evo) on my ISAR fork
>
>  https://github.com/robang74/isar
>
> * 9fd5282 - changes for a faster build using less disk space, p.1
> * 1b6ac1e - bugfix: no sstate archive obtainable, will run full task instead
> * 2cc1854 - bugfix: do_copy_boot_files was able to fail but then set -e

The patchset is changed in this way:

* 9773c8d - changes for a faster build using less disk space, p.4
(HEAD, origin/evo, origin/HEAD, evo)
* 3501d20 - changes for a faster build using less disk space, p.3
* ab0eb44 - changes for a faster build using less disk space, p.2
* 09e2bf4 - changes for a faster build using less disk space, p.1
* b9257a1 - bugfix: no sstate archive obtainable, will run full task instead
* 2cc1854 - bugfix: do_copy_boot_files was able to fail but then set -e

The second bugfix (b9257a1) has been reworked because it sometimes was
creating trouble. The first bugfix and p.1 did not change. While p.2
fixes the Felix issue on iot2050 and adopts --reflink=auto when using
'cp'. The p.3 is a straightforward version of the improvement sent
here in list by Uladzimir. While the p.4 needs to be discussed because
it is a little radical as an approach and did not bring such a change.

        patch 1         patches 1,2,3
    ------------ basic-os --------------
     8657 Mb (max)   |  4876 Mb (max)
     4118 Mb (rest)  |  4028 Mb (rest)
     3741 Mb (deb)   |  3741 Mb (deb)
      820 Mb (wic)   |   820 Mb (wic)
      579 Mb (cache) |   550 Mb (cache)
    time: 4m34s      | time: 4m24s

          patches 1-3        patches 1-4
        ------------ basic-os --------------
         4876 Mb (max)   |  4801 Mb (max)
         4028 Mb (rest)  |  3952 Mb (rest)
         3741 Mb (deb)   |  3741 Mb (deb)
          820 Mb (wic)   |   820 Mb (wic)
          550 Mb (cache) |   550 Mb (cache)
        time: 4m24s      | time: 4m16s

Best regards, R-

Re: Better way to handle apt cache needed

[Moessbauer, Felix]

On Fri, 2022-12-30 at 09:27 +0100, Roberto A. Foglietta wrote:
> On Fri, 30 Dec 2022 at 07:05, Moessbauer, Felix
> <felix.moessbauer@siemens.com> wrote:
> > 
> > On Fri, 2022-12-30 at 00:15 +0100, Roberto A. Foglietta wrote:
> > > Hi all,
> > > 
> > > I did some changes to reduce disk usage and to speed up the
> > > building.
> > > The results are quite impressing so, before everything else I am
> > > going
> > > to show you the numbers:
> > > 
> > > results before and after the changes
> > > 
> > > 43954 Mb (max)   |  8657 Mb (max)
> > > 26548 Mb (rest)  |  4118 Mb (rest)
> > >  3741 Mb (deb)   |  3741 Mb (deb)
> > >   820 Mb (wic)   |   820 Mb (wic)
> > > 11789 Mb (cache) |   579 Mb (cache)
> > > time: 8m33s      | time: 4m34s
> > > 
> > > The changes has been committed on the default branch (evo) on my
> > > ISAR
> > > fork
> > > 
> > >  https://github.com/robang74/isar
> > > 
> > > * 9fd5282 - changes for a faster build using less disk space, p.1

Just another comment - and another reminder to send the patches via the
ML:

buildchroot_install_files:append() {
    if [ -e '/usr/bin/qemu-${QEMU_ARCH}-static' ]; then
       sudo cp '/usr/bin/qemu-${QEMU_ARCH}-static'
'${BUILDCHROOT_DIR}/usr/bin/qemu-${QEMU_ARCH}-static'
       sudo ln -Pf '/usr/bin/qemu-${QEMU_ARCH}-static'
'${BUILDCHROOT_DIR}/usr/bin/qemu-${QEMU_ARCH}-static'
    fi
}

This does not work. First, this is a dynamically linked executable that
cannot be copied from the host into the buildchroot as this might break
the ABI. Second, this simply does not work:
ln: failed to create hard link '/build/tmp/work/iot2050-edgeconnect-
arm64/buildchroot-host/1.0-r0/rootfs/usr/bin/qemu-aarch64-static' =>
'/usr/bin/qemu-aarch64-static': Invalid cross-device link

Felix

> > 
> > Hi Roberto,
> > 
> > I tested your patches. In general, the implemented pattern looks
> > good
> > and also result in significantly shorter build times on virtual
> > file
> > systems.
> > 
> > But I discovered some issues on partial rebuilds. Example:
> > > ln: will not overwrite just-created
> > > '/build/tmp/deploy/buildchroot-
> > target/debian-bullseye-amd64/var/cache/apt/archives/libnet-ssleay-
> > perl_1.88-3+b1_arm64.deb' with '/build/downloads/deb/debian-
> > bullseye/libnet-ssleay-perl_1.88-3+b1_arm64.deb'
> > 
> > This happens in the do_install_npm in the npm class of meta-iot2050
> > [1].
> > It could also be an issue with an older sstate cache artifact, or
> > related to the specific implementation in the NPM class.
> > Needs further investigation.
> > 
> > [1]
> > https://github.com/siemens/meta-iot2050/blob/master/classes/npm.bbclass#L105
> 
> Hi Felix, try als with the 2nd part which do not force overwriting of
> exporting packages
> 
> 281bcdd - changes for a faster build using less disk space, p.2
> 
> I got this error in building the iot2050 project as proposed in
> README.md:
> 
> ./kas-container build kas-iot2050-example.yml:kas/opt/sdk.yml
> 
> ERROR: python-absl-0.15.0-2-r0 do_fetch: Fetcher failure: Recipe uses
> a floating tag/branch without a fixed SRCREV yet doesn't call
> bb.fetch2.get_srcrev() (use SRCPV in PV for OE).
> ERROR: Logfile of failure stored in:
> /build/tmp/work/iot2050-debian-arm64/python-absl/0.15.0-2-
> r0/temp/log.do_fetch.746
> ERROR: Task (/build/../work/meta-coral/recipes-python/python-
> absl/python-absl_0.15.0-2.bb:do_fetch)
> failed with exit code '1'
> 
> The only change that I make, is to use my ISAR and obviously I
> converted the syntax to the new bitbake 2.0
> 
> Best regards, R-

Re: Better way to handle apt cache needed

[Roberto A. Foglietta]

On Fri, 30 Dec 2022 at 14:12, Moessbauer, Felix
<felix.moessbauer@siemens.com> wrote:

>
> Just another comment - and another reminder to send the patches via the
> ML:
>
> buildchroot_install_files:append() {
>     if [ -e '/usr/bin/qemu-${QEMU_ARCH}-static' ]; then
>        sudo cp '/usr/bin/qemu-${QEMU_ARCH}-static'
> '${BUILDCHROOT_DIR}/usr/bin/qemu-${QEMU_ARCH}-static'
>        sudo ln -Pf '/usr/bin/qemu-${QEMU_ARCH}-static'
> '${BUILDCHROOT_DIR}/usr/bin/qemu-${QEMU_ARCH}-static'
>     fi
> }
>
> This does not work. First, this is a dynamically linked executable that
> cannot be copied from the host into the buildchroot as this might break
> the ABI.

The idea of copying this file is not mine but ISAR and -static could
lie but also not, depending if it is true static or just "as much as
possible".

> Second, this simply does not work:
> ln: failed to create hard link '/build/tmp/work/iot2050-edgeconnect-
> arm64/buildchroot-host/1.0-r0/rootfs/usr/bin/qemu-aarch64-static' =>
> '/usr/bin/qemu-aarch64-static': Invalid cross-device link
>

This is my fault, I am going to fix it. Thanks, R-

Re: Better way to handle apt cache needed

[Roberto A. Foglietta]

On Fri, 30 Dec 2022 at 14:33, Roberto A. Foglietta
<roberto.foglietta@gmail.com> wrote:
>
> On Fri, 30 Dec 2022 at 14:12, Moessbauer, Felix
> <felix.moessbauer@siemens.com> wrote:
>
> >
> > Just another comment - and another reminder to send the patches via the
> > ML:
> >
> > buildchroot_install_files:append() {
> >     if [ -e '/usr/bin/qemu-${QEMU_ARCH}-static' ]; then
> >        sudo cp '/usr/bin/qemu-${QEMU_ARCH}-static'
> > '${BUILDCHROOT_DIR}/usr/bin/qemu-${QEMU_ARCH}-static'
> >        sudo ln -Pf '/usr/bin/qemu-${QEMU_ARCH}-static'
> > '${BUILDCHROOT_DIR}/usr/bin/qemu-${QEMU_ARCH}-static'
> >     fi
> > }
> >
> > This does not work. First, this is a dynamically linked executable that
> > cannot be copied from the host into the buildchroot as this might break
> > the ABI.
>
> The idea of copying this file is not mine but ISAR and -static could
> lie but also not, depending if it is true static or just "as much as
> possible".

Hi Felix, it should work, considering this

$ ls -1 /usr/bin/qemu-*-static | xargs ldd 2>&1 | grep -v "not a
dynamic executable" | grep -ve "/usr/bin/qemu-.*-static:" | wc -l
0
$ ls -1 /usr/bin/qemu-*-static | wc -l
34

>
> > Second, this simply does not work:
> > ln: failed to create hard link '/build/tmp/work/iot2050-edgeconnect-
> > arm64/buildchroot-host/1.0-r0/rootfs/usr/bin/qemu-aarch64-static' =>
> > '/usr/bin/qemu-aarch64-static': Invalid cross-device link
> >
>
> This is my fault, I am going to fix it. Thanks, R-

rebased and updated. What about p.4? What is your opinion

Best regards, R-

Re: Better way to handle apt cache needed

[Roberto A. Foglietta]

[-- Attachment #1: Type: text/plain, Size: 738 bytes --]

Hi all,

 sorry for the image but some numbers should be displayed in a proper
form. I continued to work on the building optimization of ISAR and the
results are quite interesting. Almost 2x with some great achievements
in some specific areas like cache reduction (22x) or in some
particular cases like using a little of the repository downloaded
which brings to a disk usage reduced by 11x times. About building
time, the ratios are a little under-estimated because network
downloads for debootstrap and apt update take a fixed time that cannot
be reduced. My first tests indicate that also custom debian packages
and repackaged .deb are correctly deployed into the system despite the
changes. More tests are welcome.

 Best regards, R-

[-- Attachment #2: isar-building-performance-gains.png --]
[-- Type: image/png, Size: 40824 bytes --]

Re: Better way to handle apt cache needed

[Roberto A. Foglietta]

On Sat, 31 Dec 2022 at 09:59, Roberto A. Foglietta
<roberto.foglietta@gmail.com> wrote:
>
> Hi all,
>
>  About building
> time, the ratios are a little under-estimated because network
> downloads for debootstrap and apt update take a fixed time that cannot
> be reduced.

that CANNOT be reduced - not so fast buddy!
Saved 47s on my 7Mbit/s internet connection:

 4588 Mb (max)
 3731 Mb (rest)
 3887 Mb (deb)
  820 Mb (wic)
  522 Mb (cache)
time: 3m30s (2.44x)

Changes used:

* d11e783 - bootstrap im/export apt update lists to fasten the whole procedure
* a96b047 - bootstrap uset proxy env vars before bootstrap, unsupported yet
* 9d5f3e5 - deb-dl-dir deb_lists_dir_im/export added + rationalisation

Happy new year in two hours, R-

Re: Better way to handle apt cache needed

[Henning Schild]

Am Wed, 28 Dec 2022 10:02:13 +0100
schrieb "Moessbauer, Felix (T CED INW-CN)"
<felix.moessbauer@siemens.com>:

> Hi,
> 
> when working with builds that have both many recipes, as well as many
> build dependencies, disk usage explodes during the build. As both
> preconditions somehow correspond, this results in a quadratic disc
> consumption in the number of tasks during the build.
> 
> The root cause for that behavior is the apt cache
> (deb_dl_dir_(import|export)), that copies all previously downloaded
> apt packages into the WORKDIR of each (bitbake) package.
> Given, that a common apt-cache is around 2GB and 8 tasks are run in
> parallel, this gives already 16GB for the tasks, and 7 * 2GB for the
> buildchroots (host and target), in total ~30GB.
>
> In one of my projects, we have to work with huge debian packages,
> leading to apt-cache sizes around 20GB. As these projects usually also
> have to be built on big machines with many cores, you easily get 500GB
> of required scratch disk space + a lot of disc accesses for the copy,
> making it basically impossible to build the project except by limiting
> the number of tasks that run in parallel.
> 
> Given that, we should really think about a way to get the disc
> consumption back to a linear level. Ideally, we would only use
> symlinks or maybe hardlinks to deduplicate. Another option would be
> to use the POSIX atomicity guarantees by just renaming packages when
> inserting into the cache.
> 
> Anyways, we need a better solution.
> Putting Henning as the author of that logic in CC.

Thanks for bringing that up. I did not yet realize that problem but it
sounds very real and valid.

That thread here is already very long just looking at it and not having
read much of it.

All i can say in my defense is that sbuild is likely what took as from
linear to multiplication. So technically i have nothing to do with any
of this ;).

Henning

> Best regards,
> Felix

Re: Better way to handle apt cache needed

[Uladzimir Bely]

In the email from Wednesday, 28 December 2022 12:02:13 +03 user Moessbauer, 
Felix wrote:
> Hi,
> 
> when working with builds that have both many recipes, as well as many
> build dependencies, disk usage explodes during the build. As both
> preconditions somehow correspond, this results in a quadratic disc
> consumption in the number of tasks during the build.
> 
> The root cause for that behavior is the apt cache
> (deb_dl_dir_(import|export)), that copies all previously downloaded apt
> packages into the WORKDIR of each (bitbake) package.
> Given, that a common apt-cache is around 2GB and 8 tasks are run in
> parallel, this gives already 16GB for the tasks, and 7 * 2GB for the
> buildchroots (host and target), in total ~30GB.
> 
> In one of my projects, we have to work with huge debian packages,
> leading to apt-cache sizes around 20GB. As these projects usually also
> have to be built on big machines with many cores, you easily get 500GB
> of required scratch disk space + a lot of disc accesses for the copy,
> making it basically impossible to build the project except by limiting
> the number of tasks that run in parallel.
> 
> Given that, we should really think about a way to get the disc
> consumption back to a linear level. Ideally, we would only use symlinks
> or maybe hardlinks to deduplicate. Another option would be to use the
> POSIX atomicity guarantees by just renaming packages when inserting
> into the cache.
> 
> Anyways, we need a better solution.
> Putting Henning as the author of that logic in CC.
> 
> Best regards,
> Felix

Hi all

I'd like just to mention unfinished patchset that I was working on earlier. It 
was last sent to maillist as `[PATCH v3 0/5] Improving base-apt usage PoC`.

The idea was to predownload all possible package dependencies to 'base-apt' 
repo first and use it (as "file:///path/to/base-apt") in sources list.

In this case (e.g. "file://..." source), as far as I remember, apt doesn't 
need packages to be "downloaded" to /var/cache/apt and just use them directly.

Since that time, of course, Isar changed (for example, now we have host/target 
splitted base-apt), so the patchset need to be updated. Also it requires some 
cleanup and improvements (technically, we don't need $DL_DIR/deb at all, since 
we use local base-apt repo).

Anyway, such kind of local-repo-based approach would be a good solution for 
high disk usage problem.

-- 
Uladzimir Bely

Re: Better way to handle apt cache needed

[Roberto A. Foglietta]

On Thu, 5 Jan 2023 at 07:31, Uladzimir Bely <ubely@ilbers.de> wrote:

> Anyway, such kind of local-repo-based approach would be a good solution for
> high disk usage problem.

Hi Uladzimir,

a local repo sounds like a great idea and I gave it a thought too.
However, the repo should be prepared, it is not enough that it
contains the .deb files like in the /var/cache/apt. Fortunately, the
preparation of the repo does not seem a great deal but it should be
repeated at every import / export and this might impair the timing
performance of the building.

Best regards, R-

Re: Better way to handle apt cache needed

[Roberto A. Foglietta]

[-- Attachment #1.1: Type: text/plain, Size: 3000 bytes --]

On Sat, 31 Dec 2022 at 09:59, Roberto A. Foglietta <
roberto.foglietta@gmail.com> wrote:
>
> Hi all,
>
>  sorry for the image but some numbers should be displayed in a proper
> form. I continued to work on the building optimization of ISAR and the
> results are quite interesting. Almost 2x with some great achievements
> in some specific areas like cache reduction (22x) or in some
> particular cases like using a little of the repository downloaded
> which brings to a disk usage reduced by 11x times. About building
> time, the ratios are a little under-estimated because network
> downloads for debootstrap and apt update take a fixed time that cannot
> be reduced. My first tests indicate that also custom debian packages
> and repackaged .deb are correctly deployed into the system despite the
> changes. More tests are welcome.
>

Dear all,

my personal fork was broken in some ways for some time but today, I
completed the first round of rebasing and finally it is up again.
These are the numbers, formatted in HTML for a nice reading and an image
attached as well.

>From these numbers the first thing we notice is that the size of the deb
cache
does not matter at all because 10x more is not slower at all. This makes
per-
fectly sense because linking 385 or 1260 does not make such a difference.

          full debs           minimal           rebnext
       ------------ basic-os -------------- ==============
        3498 Mb (max)   |  3489 Mb (max)     1.00x   1.00x
        2588 Mb (rest)  |  2587 Mb (rest)    1.00x   1.00x
        3417 Mb (deb)   |   364 Mb (deb)     9.39x
         814 Mb (wic)   |   814 Mb (wic)      -
         273 Mb (cache) |   273 Mb (cache)   1.00x
       time: 3m09s      | time: 3m10s        2.58x

Compared to the original the performances increase is impressive, in both
cases:

          original           cherries + schroot + rebase
       ------------ basic-os -------------- ==============
       43954 Mb (max)   |  3498 Mb (max)    12.57x  16.07x
       26548 Mb (rest)  |  2588 Mb (rest)   10.26x  14.50x
        3741 Mb (deb)   |  3417 Mb (deb)      -
         820 Mb (wic)   |   814 Mb (wic)      -
       11789 Mb (cache) |   273 Mb (cache)  43.18x
       time: 8m33s      | time: 3m09s        2.71x

          original           cherries + schroot + rebase
       ------------ complete ------------- ===============
       52507 Mb (max)   | 28606 Mb (max)     1.84x   2.23x
       43311 Mb (rest)  | 19413 Mb (rest)    2.23x   3.33x
        3741 Mb (deb)   |  3417 Mb (deb)      -
        9159 Mb (wic)   |  9155 Mb (wic)      -
       11799 Mb (cache) |   283 Mb (cache)  41.69x
       time: 20m13s     | time: 11m44s       1.72x

Considering that the local .deb cache growing with the time because new
updates
will be added, and comparing the performance gains between the basic-os and
the
complete image, we can say that on the long run also the complete case will
reach the 2x of performance at least in time building.

Best regards, R.

[-- Attachment #1.2: Type: text/html, Size: 3648 bytes --]

[-- Attachment #2: rebnext-performances-gain.png --]
[-- Type: image/png, Size: 54571 bytes --]

Re: Better way to handle apt cache needed

[Roberto A. Foglietta]

On Mon, 9 Jan 2023 at 09:12, Roberto A. Foglietta
<roberto.foglietta@gmail.com> wrote:
>
>
>
> On Sat, 31 Dec 2022 at 09:59, Roberto A. Foglietta <roberto.foglietta@gmail.com> wrote:
> >
> > Hi all,
> >
> >  sorry for the image but some numbers should be displayed in a proper
> > form. I continued to work on the building optimization of ISAR and the
> > results are quite interesting. Almost 2x with some great achievements
> > in some specific areas like cache reduction (22x) or in some
> > particular cases like using a little of the repository downloaded
> > which brings to a disk usage reduced by 11x times. About building
> > time, the ratios are a little under-estimated because network
> > downloads for debootstrap and apt update take a fixed time that cannot
> > be reduced. My first tests indicate that also custom debian packages
> > and repackaged .deb are correctly deployed into the system despite the
> > changes. More tests are welcome.
> >
>
> Dear all,
>
> my personal fork was broken in some ways for some time but today, I completed the first round of rebasing and finally it is up again.
> These are the numbers, formatted in HTML for a nice reading and an image attached as well.
>
> From these numbers the first thing we notice is that the size of the deb cache
> does not matter at all because 10x more is not slower at all. This makes per-
> fectly sense because linking 385 or 1260 does not make such a difference.
>
>           full debs           minimal           rebnext
>        ------------ basic-os -------------- ==============
>         3498 Mb (max)   |  3489 Mb (max)     1.00x   1.00x
>         2588 Mb (rest)  |  2587 Mb (rest)    1.00x   1.00x
>         3417 Mb (deb)   |   364 Mb (deb)     9.39x
>          814 Mb (wic)   |   814 Mb (wic)      -
>          273 Mb (cache) |   273 Mb (cache)   1.00x
>        time: 3m09s      | time: 3m10s        2.58x
>
> Compared to the original the performances increase is impressive, in both cases:
>
>           original           cherries + schroot + rebase
>        ------------ basic-os -------------- ==============
>        43954 Mb (max)   |  3498 Mb (max)    12.57x  16.07x
>        26548 Mb (rest)  |  2588 Mb (rest)   10.26x  14.50x
>         3741 Mb (deb)   |  3417 Mb (deb)      -
>          820 Mb (wic)   |   814 Mb (wic)      -
>        11789 Mb (cache) |   273 Mb (cache)  43.18x
>        time: 8m33s      | time: 3m09s        2.71x
>
>           original           cherries + schroot + rebase
>        ------------ complete ------------- ===============
>        52507 Mb (max)   | 28606 Mb (max)     1.84x   2.23x
>        43311 Mb (rest)  | 19413 Mb (rest)    2.23x   3.33x
>         3741 Mb (deb)   |  3417 Mb (deb)      -
>         9159 Mb (wic)   |  9155 Mb (wic)      -
>        11799 Mb (cache) |   283 Mb (cache)  41.69x
>        time: 20m13s     | time: 11m44s       1.72x
>
> Considering that the local .deb cache growing with the time because new updates
> will be added, and comparing the performance gains between the basic-os and the
> complete image, we can say that on the long run also the complete case will
> reach the 2x of performance at least in time building.

Results has been publish and the 1st round rebase branch has set as default

https://github.com/robang74/isar/blob/rebnext/doc/build-optim.md

Best regards, R-

Re: Better way to handle apt cache needed

[Roberto A. Foglietta]

[-- Attachment #1.1: Type: text/plain, Size: 2242 bytes --]

Hi all,

Thanks to Uladzimir who pointed me out about the weirdness of ln -P in
sbuild
and other changes that I made, nvpriv7 is clearly faster than npriv6. Here
below the benchmarks compared with the original based on robang74:next.

Thanks to Uladzimir who pointed me out about the weirdness of ln -P in
sbuild
and other changes that I made, nvpriv7 is clearly faster than npriv6. Here
below the benchmarks compared with the original based on robang74:next.

        devel2 (public) + npriv7 (private)
       ------------------------------------
            fresh              cache
       ------------ basic-os --------------
        3745 Mb (max)   |  3182 Mb (max)
        2849 Mb (rest)  |  2289 Mb (rest)
        3596 Mb (deb)   |  3596 Mb (deb)
         806 Mb (wic)   |   806 Mb (wic)
         551 Mb (cache) |   551 Mb (cache)
       time:  2m39s     | time:    30s       5.30x
       ------------ original ---------------------
       time:  7m58s     | time:  2m17s       3.48x
              3.00x     |        4.57x
                        |
       ------------ complete --------------
       33197 Mb (max)   | 28209 Mb (max)
       23964 Mb (rest)  | 18982 Mb (rest)
        3596 Mb (deb)   |  3596 Mb (deb)
        9143 Mb (wic)   |  9139 Mb (wic)
        4916 Mb (cache) |  4916 Mb (cache)
       time: 12m45s     | time:  4m11s       3.05x
       ------------ original ---------------------
       time: 21m17s     | time:  8m28s       2.51x
              1.67x     |        2.02x

About the complete image, we need to consider some fixed times like these:

* time to produce complete wic image: 2m23s
* time to compile nvidia drivers: 2m8s

Because these times cannot be shrink as part of the building process then
we can
subtract that times (for the cache just producing the image because the
drivers
are cached) and revise the benchmarks as following:

            fresh              cache
       ------------ complete --------------
       time:  8m14s     | time:  1m48s       4.57x
       ------------ original ---------------------
       time: 16m46s     | time:  6m05s       2.76x
              2.03x     |        3.38x

Even in the worst case, the building performances has been increased by 2x


Best regards, R-

[-- Attachment #1.2: Type: text/html, Size: 2763 bytes --]

[-- Attachment #2: variance-among-projects-npriv7.png --]
[-- Type: image/png, Size: 50186 bytes --]

Re: Better way to handle apt cache needed

[Roberto A. Foglietta]

On Thu, 19 Jan 2023 at 19:08, Roberto A. Foglietta
<roberto.foglietta@gmail.com> wrote:
>
> Hi all,
>
> Thanks to Uladzimir who pointed me out about the weirdness of ln -P in sbuild
> and other changes that I made, nvpriv7 is clearly faster than npriv6. Here
> below the benchmarks compared with the original based on robang74:next.
>
> Thanks to Uladzimir who pointed me out about the weirdness of ln -P in sbuild
> and other changes that I made, nvpriv7 is clearly faster than npriv6. Here
> below the benchmarks compared with the original based on robang74:next.
>
>         devel2 (public) + npriv7 (private)
>        ------------------------------------
>             fresh              cache
>        ------------ basic-os --------------
>         3745 Mb (max)   |  3182 Mb (max)
>         2849 Mb (rest)  |  2289 Mb (rest)
>         3596 Mb (deb)   |  3596 Mb (deb)
>          806 Mb (wic)   |   806 Mb (wic)
>          551 Mb (cache) |   551 Mb (cache)
>        time:  2m39s     | time:    30s       5.30x

This 2m39s is wrong: it refers to a reduced image that I created for
test and I committed some hours ago

https://github.com/robang74/isar-nvidia-debian/commit/7ea25b486484b2f675bc1a94098dd22fb08594f0

The real number is 3m6s for the current npriv.8 - while 30s has been
reduced at 28s. Sorry for the mistake, my fault.

>        ------------ original ---------------------
>        time:  7m58s     | time:  2m17s       3.48x
>               3.00x     |        4.57x
>                         |
>        ------------ complete --------------
>        33197 Mb (max)   | 28209 Mb (max)
>        23964 Mb (rest)  | 18982 Mb (rest)
>         3596 Mb (deb)   |  3596 Mb (deb)
>         9143 Mb (wic)   |  9139 Mb (wic)
>         4916 Mb (cache) |  4916 Mb (cache)
>        time: 12m45s     | time:  4m11s       3.05x
>        ------------ original ---------------------
>        time: 21m17s     | time:  8m28s       2.51x
>               1.67x     |        2.02x
>
> About the complete image, we need to consider some fixed times like these:
>
> * time to produce complete wic image: 2m23s
> * time to compile nvidia drivers: 2m8s
>
> Because these times cannot be shrink as part of the building process then we can
> subtract that times (for the cache just producing the image because the drivers
> are cached) and revise the benchmarks as following:
>
>             fresh              cache
>        ------------ complete --------------
>        time:  8m14s     | time:  1m48s       4.57x
>        ------------ original ---------------------
>        time: 16m46s     | time:  6m05s       2.76x
>               2.03x     |        3.38x
>
> Even in the worst case, the building performances has been increased by 2x
>
> Best regards, R-



-- 
Roberto A. Foglietta
+49.176.274.75.661
+39.349.33.30.697

Re: Better way to handle apt cache needed

[Roberto A. Foglietta]

[-- Attachment #1.1: Type: text/plain, Size: 1916 bytes --]

On Wed, 25 Jan 2023 at 05:48, Roberto A. Foglietta <
roberto.foglietta@gmail.com> wrote:

> On Thu, 19 Jan 2023 at 19:08, Roberto A. Foglietta



SSTATE CACHE REWORK + APT CACHE MOUNT BIND
==========================================

These are the results in the first development version that works in every
condition. It carries on a lot of debugging code and it is NOT fully
optimised
both under the code-review and the performances.
Nevertheless, it still improved:

        devel3 (public) + npriv11 (private)
       ------------------------------------
            fresh              cache
       ------------ basic-os --------------
        3949 Mb (max)   |  3176 Mb (max)
        2968 Mb (rest)  |  2277 Mb (rest)
        4281 Mb (deb)   |  4281 Mb (deb)
         803 Mb (wic)   |   802 Mb (wic) <-- cache!
         520 Mb (cache) |   520 Mb (cache)
       time:  3m28s     | time:    20s      10.40x
       ------------ original ---------------------
       time:  7m58s     | time:  2m17s
              2.30x              6.85x

        devel3 (public) + npriv11 (private)
       ------------------------------------
            fresh              cache
       ------------ complete --------------
       32466 Mb (max)   | 27741 Mb (max)
       23298 Mb (rest)  | 18657 Mb (rest)
        4281 Mb (deb)   |  4281 Mb (deb)
        8989 Mb (wic)   |  8987 Mb (wic) <-- cache!
        4398 Mb (cache) |  4398 Mb (cache)
       time: 12m19s     | time:  3m24s       3.62x
       ------------ original ---------------------
       time: 21m17s     | time:  8m28s
              1.73x              2.49x

The mount bind approach is completely general but not yet used for apt
lists but
just for the .deb packages. Moreover, the rootfs tar options (--exclude/s)
are
too extreme and the cached build is smaller than the fresh build. This
obviously
destroys the reproducibility but it is just a single point change.

[-- Attachment #1.2: Type: text/html, Size: 2696 bytes --]

[-- Attachment #2: performances-report-v4.txt --]
[-- Type: text/plain, Size: 1785 bytes --]



SSTATE CACHE REWORK + APT CACHE MOUNT BIND 
==========================================

These are the results in the first development version that works in every
condition. It carries on a lot of debugging code and it is NOT fully optimised
both under the code-review and the performances.
Nevertheless, it still improved:

        devel3 (public) + npriv11 (private)
       ------------------------------------
            fresh              cache
       ------------ basic-os --------------
        3949 Mb (max)   |  3176 Mb (max)
        2968 Mb (rest)  |  2277 Mb (rest)
        4281 Mb (deb)   |  4281 Mb (deb)
         803 Mb (wic)   |   802 Mb (wic) <-- cache!
         520 Mb (cache) |   520 Mb (cache)
       time:  3m28s     | time:    20s      10.40x
       ------------ original ---------------------
       time:  7m58s     | time:  2m17s
              2.30x              6.85x

        devel3 (public) + npriv11 (private)
       ------------------------------------
            fresh              cache
       ------------ complete --------------
       32466 Mb (max)   | 27741 Mb (max)
       23298 Mb (rest)  | 18657 Mb (rest)
        4281 Mb (deb)   |  4281 Mb (deb)
        8989 Mb (wic)   |  8987 Mb (wic) <-- cache!
        4398 Mb (cache) |  4398 Mb (cache)
       time: 12m19s     | time:  3m24s       3.62x
       ------------ original ---------------------
       time: 21m17s     | time:  8m28s            
              1.73x              2.49x

The mount bind approach is completely general but not yet used for apt lists but
just for the .deb packages. Moreover, the rootfs tar options (--exclude/s) are
too extreme and the cached build is smaller than the fresh build. This obviously
destroys the reproducibility but it is just a single point change.

Re: Better way to handle apt cache needed

[Roberto A. Foglietta]

[-- Attachment #1.1: Type: text/plain, Size: 3422 bytes --]

On Fri, 10 Feb 2023 at 17:05, Roberto A. Foglietta <
roberto.foglietta@gmail.com> wrote:

> On Wed, 25 Jan 2023 at 05:48, Roberto A. Foglietta <
> roberto.foglietta@gmail.com> wrote:
>
>> On Thu, 19 Jan 2023 at 19:08, Roberto A. Foglietta
>
>
SSTATE CACHE REWORK + APT CACHE MOUNT BIND
==========================================

Debug code is still in place but the APT cache and lists are offered by
binding.
Binding is the favourite way to offer APT cache and repositories lists but
not
with sbuild because the lock contention is not supported with binding. It is
using soft links but in an improved declination.

        devel4 (public) + npriv12 (private)
       ------------------------------------
            fresh              cache
       ------------ complete --------------
       32417 Mb (max)   | 27740 Mb (max)
       23294 Mb (rest)  | 18657 Mb (rest)
        4283 Mb (deb)   |  4283 Mb (deb)
        8989 Mb (wic)   |  8987 Mb (wic) <-- cache!
        4398 Mb (cache) |  4398 Mb (cache)
       time: 12m01s     | time:  2m30s [*]   4.81x
       ------------ original ---------------------
       time: 21m17s     | time:  8m28s
              1.77x              3.39x

        devel4 (public) + npriv12 (private)
       ------------------------------------
            fresh              cache
       ------------ basic-os --------------
        3946 Mb (max)   |  3176 Mb (max)
        2976 Mb (rest)  |  2278 Mb (rest)
        4283 Mb (deb)   |  4283 Mb (deb)
         803 Mb (wic)   |   802 Mb (wic) <-- cache!
         520 Mb (cache) |   520 Mb (cache)
       time:  3m05s     | time:    20s       9.25x
       ------------ original ---------------------
       time:  7m58s     | time:  2m17s
              2.58x              6.85x

These numbers show that Ilbers' ISAR improved a lot during the time

    original, fresh                     21m17s
    complete, next, devel3, fresh       16m49s   1.27x
    complete, next.plus, devel3, fresh  15m49s   1.35x
    complete, npriv.12, devel4, fresh   12m01s   1.77x

    original, cache                      8m28s
    complete, next, devel3, cache        5m15s   1.61x
    complete, next.plus, devel3, cache   3m18s   2.57x
    complete, npriv.12, devel4, cache    2m30s   3.39x

For completeness: next.plus is the next with all the patches applied from
the
mailing list, all those patches that are supposed to improve the building
speed.

Considering that the nvidia-modules compilation takes 2m4s and producing
the wic
image takes 2m8s in the complete build but these activities are not part of
the
optimisation, we have these numbers:

    original, fresh                     17m05s
    complete, next, devel3, fresh       12m37s   1.35x
    complete, next.plus, devel3, fresh  11m37s   1.48x
    complete, npriv.12, devel4, fresh    7m49s   2.19x

    original, cache                      6m20s
    complete, next, devel3, cache        3m07s   2.03x
    complete, next.plus, devel3, cache   1m10s   5.43x
    complete, npriv.12, devel4, cache    0m22s  17.27x

The last number 22s is coherent with the basic-os cached building time.

[*] after 2m the complete fresh build, SSD needs to relax internally which
is
    fully acceptable in a real-case scenario. The original time values are
still
    correct because at the time they have been taken there was not a script
that
    immediately after the complete build was started the cached building.

[-- Attachment #1.2: Type: text/html, Size: 4845 bytes --]

[-- Attachment #2: performances-report-v5.txt --]
[-- Type: text/plain, Size: 3273 bytes --]


SSTATE CACHE REWORK + APT CACHE MOUNT BIND 
==========================================

Debug code is still in place but the APT cache and lists are offered by binding.
Binding is the favourite way to offer APT cache and repositories lists but not
with sbuild because the lock contention is not supported with binding. It is
using soft links but in an improved declination.

        devel4 (public) + npriv12 (private)
       ------------------------------------
            fresh              cache
       ------------ complete --------------
       32417 Mb (max)   | 27740 Mb (max)
       23294 Mb (rest)  | 18657 Mb (rest)
        4283 Mb (deb)   |  4283 Mb (deb)
        8989 Mb (wic)   |  8987 Mb (wic) <-- cache!
        4398 Mb (cache) |  4398 Mb (cache)
       time: 12m01s     | time:  2m30s [*]   4.81x
       ------------ original ---------------------
       time: 21m17s     | time:  8m28s            
              1.77x              3.39x

        devel4 (public) + npriv12 (private)
       ------------------------------------
            fresh              cache
       ------------ basic-os --------------
        3946 Mb (max)   |  3176 Mb (max)
        2976 Mb (rest)  |  2278 Mb (rest)
        4283 Mb (deb)   |  4283 Mb (deb)
         803 Mb (wic)   |   802 Mb (wic) <-- cache!
         520 Mb (cache) |   520 Mb (cache)
       time:  3m05s     | time:    20s       9.25x
       ------------ original ---------------------
       time:  7m58s     | time:  2m17s
              2.58x              6.85x

These numbers show that Ilbers' ISAR improved a lot during the time:

    original, fresh                     21m17s
    complete, next, devel3, fresh       16m49s   1.27x
    complete, next.plus, devel3, fresh  15m49s   1.35x
    complete, npriv.12, devel4, fresh   12m01s   1.77x
	
    original, cache                      8m28s
    complete, next, devel3, cache        5m15s   1.61x
    complete, next.plus, devel3, cache   3m18s   2.57x
    complete, npriv.12, devel4, cache    2m30s   3.39x

For completeness: next.plus is the next with all the patches applied from the 
mailing list, all those patches that are supposed to improve the building speed.
While devel3 and devel4 differ only about how they inherit custom classes.

Considering that the nvidia-modules compilation takes 2m4s and producing the wic
image takes 2m8s in the complete build but these activities are not part of the
optimisation, we have these numbers:

    original, fresh                     17m05s
    complete, next, devel3, fresh       12m37s   1.35x
    complete, next.plus, devel3, fresh  11m37s   1.48x
    complete, npriv.12, devel4, fresh    7m49s   2.19x
	
    original, cache                      6m20s
    complete, next, devel3, cache        3m07s   2.03x
    complete, next.plus, devel3, cache   1m10s   5.43x
    complete, npriv.12, devel4, cache    0m22s  17.27x

The last number 22s is coherent with the basic-os cached building time.

[*] after 2m the complete fresh build, SSD needs to relax internally which is
    fully acceptable in a real-case scenario. The original time values are still
    correct because at the time they have been taken there was not a script that
    immediately after the complete build was started the cached building.

Re: Better way to handle apt cache needed

[Roberto A. Foglietta]

[-- Attachment #1: Type: text/plain, Size: 561 bytes --]

On Tue, 14 Feb 2023 at 11:01, Roberto A. Foglietta <
roberto.foglietta@gmail.com> wrote:

> On Fri, 10 Feb 2023 at 17:05, Roberto A. Foglietta <
> roberto.foglietta@gmail.com> wrote:
>
>> On Wed, 25 Jan 2023 at 05:48, Roberto A. Foglietta <
>> roberto.foglietta@gmail.com> wrote:
>>
>>> On Thu, 19 Jan 2023 at 19:08, Roberto A. Foglietta
>>
>>
The branch next.plus used for tests is on-line with tag results-v5 for
future reference and quick adoption in kas.yml

https://github.com/ilbers/isar/commit/b755b520519a0e45e2c39bd44aeb03e5f8c23d38

 Best regards, R-

[-- Attachment #2: Type: text/html, Size: 1551 bytes --]