Re: Reliability and build time improvements with apt-cacher-ng

["'Alexander Heinisch' via isar-users" <isar-users@googlegroups.com>]

[-- Attachment #1.1: Type: text/plain, Size: 3989 bytes --]

Hi Felix!

> I just played around with apt-cacher-ng to circumvent the throttling
> issues on snapshots.d.o, and got some impressive results:

We just migrated our gitlab-runners (for a downstream project based on 
isar-cip-core) to a new build server.
The project builds 4 images (all for amd64 arch) whereas 2 of them are done 
in a multiconfig setup using the isar-installer.
In total the complete build fetches 423MB of apt packages from 
snapshot-cloudflare.debian.org

From an initial build time of ~12min (~35min on the old build server :-)) 
we could reduce build times (building all 4 images) down to ~7min using 
apt-cacher-ng.

> First of all, it is easy to configure ISAR to use the apt-cacher-ng, by
> just using the DISTRO_APT_PREMIRRORS:

Yes, we just passed `DISTRO_APT_PREMIRRORS` to our kas yaml like:

```
env:
  DISTRO_APT_PREMIRRORS: ""
```

> DISTRO_APT_SNAPSHOT_PREMIRROR = "deb.debian.org/(.*)
> localhost:3142/snapshot.debian.org/archive/\1/${APT_SNAPSHOT_DATE}/\n"

Although, we are using snapshot versions of the packages, we don't 
use `DISTRO_APT_SNAPSHOT_PREMIRROR`  as of now, but will switch soon!
 
> Then, an apt-cacher-ng instance needs to be started on the host.
> When running inside the kas container, just use the external host IP,
> or forward port 3142 into the container. While the build is running,
> you can monitor the caching statistics under
> http://localhost:3142/acng-report.html.
 
So for local builds we installed `apt install apt-cacher-ng` and
```
export 
DISTRO_APT_PREMIRRORS="snapshot-cloudflare.debian.org/archive/debian/(.*) 
localhost:3142/snapshot-cloudflare.debian.org/archive/debian/20240311/\n \
                              
snapshot-cloudflare.debian.org/archive/debian-security/(.*) 
localhost:3142/snapshot-cloudflare.debian.org/archive/debian-security/20240311/\n"
```
does the job!

> Second, the cache directory can - in theory - be cached in a CI as
> well, so subsequent jobs can use it. Then, even less load is applied
> onto snapshots.d.o. Even when using standard debian mirrors, this helps
> as usually all ISAR build jobs run in parallel, hence do not share
> already downloaded packages via ISARs internal cache.

For the build server we added a dummy interface to bind on and used 
something like:

```
image: ghcr.io/siemens/kas/kas-isar:4.0
variables:
  APT_CACHE: 192.168.42.1:3142
script:
  - export 
DISTRO_APT_PREMIRRORS="snapshot-cloudflare.debian.org/archive/debian/(.*) 
${APT_CACHE}/snapshot-cloudflare.debian.org/archive/debian/20240311/\n 
snapshot-cloudflare.debian.org/archive/debian-security/(.*) 
${APT_CACHE}/snapshot-cloudflare.debian.org/archive/debian-security/20240311/\n"
  - kas build ${TARGET}
```
in our CI pipeline.
 
> Currently I'm playing around with adding a built-in apt-cacher-ng into
> kas and the kas-container. By that, all users can instantly profit from
> the speedup. For CI systems, an ambient caching service (e.g. via
> gitlabs "services") would probably make more sense, but is also more
> complicated to configure.

Usage with kas-container script (from cip-core) did not work for me.
Since the handling of `DISTRO_APT_PREMIRRORS` as well as `--runtime-args` 
leads to `docker: invalid reference format.` once spaces are contained,
or, if escaped, the parser used in `isar-bootstrap.inc` could not split it 
correctly.
(We didn't dig too deep into that, since most of us use kas natively.)
For completeness: We are using:
isar-cip-core @ 95e952569b59602dd05f4138de9602a5b0398f0a
isar @ 3d863aad5e12c7bef382cb6fb3f1f5e18b236156

> I'm happy to hear your thoughts.

Hope that helps :-)

BR Alexander

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/a8e8e8d0-1fa3-4062-a7de-4783808bbc87n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 4709 bytes --]