Re: Better way to handle apt cache needed

["Roberto A. Foglietta" <roberto.foglietta@gmail.com>]

[-- Attachment #1.1: Type: text/plain, Size: 1916 bytes --]

On Wed, 25 Jan 2023 at 05:48, Roberto A. Foglietta <
roberto.foglietta@gmail.com> wrote:

> On Thu, 19 Jan 2023 at 19:08, Roberto A. Foglietta



SSTATE CACHE REWORK + APT CACHE MOUNT BIND
==========================================

These are the results in the first development version that works in every
condition. It carries on a lot of debugging code and it is NOT fully
optimised
both under the code-review and the performances.
Nevertheless, it still improved:

        devel3 (public) + npriv11 (private)
       ------------------------------------
            fresh              cache
       ------------ basic-os --------------
        3949 Mb (max)   |  3176 Mb (max)
        2968 Mb (rest)  |  2277 Mb (rest)
        4281 Mb (deb)   |  4281 Mb (deb)
         803 Mb (wic)   |   802 Mb (wic) <-- cache!
         520 Mb (cache) |   520 Mb (cache)
       time:  3m28s     | time:    20s      10.40x
       ------------ original ---------------------
       time:  7m58s     | time:  2m17s
              2.30x              6.85x

        devel3 (public) + npriv11 (private)
       ------------------------------------
            fresh              cache
       ------------ complete --------------
       32466 Mb (max)   | 27741 Mb (max)
       23298 Mb (rest)  | 18657 Mb (rest)
        4281 Mb (deb)   |  4281 Mb (deb)
        8989 Mb (wic)   |  8987 Mb (wic) <-- cache!
        4398 Mb (cache) |  4398 Mb (cache)
       time: 12m19s     | time:  3m24s       3.62x
       ------------ original ---------------------
       time: 21m17s     | time:  8m28s
              1.73x              2.49x

The mount bind approach is completely general but not yet used for apt
lists but
just for the .deb packages. Moreover, the rootfs tar options (--exclude/s)
are
too extreme and the cached build is smaller than the fresh build. This
obviously
destroys the reproducibility but it is just a single point change.

[-- Attachment #1.2: Type: text/html, Size: 2696 bytes --]

[-- Attachment #2: performances-report-v4.txt --]
[-- Type: text/plain, Size: 1785 bytes --]



SSTATE CACHE REWORK + APT CACHE MOUNT BIND 
==========================================

These are the results in the first development version that works in every
condition. It carries on a lot of debugging code and it is NOT fully optimised
both under the code-review and the performances.
Nevertheless, it still improved:

        devel3 (public) + npriv11 (private)
       ------------------------------------
            fresh              cache
       ------------ basic-os --------------
        3949 Mb (max)   |  3176 Mb (max)
        2968 Mb (rest)  |  2277 Mb (rest)
        4281 Mb (deb)   |  4281 Mb (deb)
         803 Mb (wic)   |   802 Mb (wic) <-- cache!
         520 Mb (cache) |   520 Mb (cache)
       time:  3m28s     | time:    20s      10.40x
       ------------ original ---------------------
       time:  7m58s     | time:  2m17s
              2.30x              6.85x

        devel3 (public) + npriv11 (private)
       ------------------------------------
            fresh              cache
       ------------ complete --------------
       32466 Mb (max)   | 27741 Mb (max)
       23298 Mb (rest)  | 18657 Mb (rest)
        4281 Mb (deb)   |  4281 Mb (deb)
        8989 Mb (wic)   |  8987 Mb (wic) <-- cache!
        4398 Mb (cache) |  4398 Mb (cache)
       time: 12m19s     | time:  3m24s       3.62x
       ------------ original ---------------------
       time: 21m17s     | time:  8m28s            
              1.73x              2.49x

The mount bind approach is completely general but not yet used for apt lists but
just for the .deb packages. Moreover, the rootfs tar options (--exclude/s) are
too extreme and the cached build is smaller than the fresh build. This obviously
destroys the reproducibility but it is just a single point change.