Hi Felix!
> I just played around with apt-cacher-ng to circumvent the throttling
> issues on snapshots.d.o, and got some impressive results:
We just migrated our gitlab-runners (for a downstream project based on isar-cip-core) to a new build server.
The project builds 4 images (all for amd64 arch) whereas 2 of them are done in a multiconfig setup using the isar-installer.
In total the complete build fetches 423MB of apt packages from snapshot-cloudflare.debian.org
From an initial build time of ~12min (~35min on the old build server :-)) we could reduce build times (building all 4 images) down to ~7min using apt-cacher-ng.
> First of all, it is easy to configure ISAR to use the apt-cacher-ng, by
> just using the DISTRO_APT_PREMIRRORS:
Yes, we just passed `DISTRO_APT_PREMIRRORS` to our kas yaml like:
```
env:
DISTRO_APT_PREMIRRORS: ""
```
> DISTRO_APT_SNAPSHOT_PREMIRROR = "deb.debian.org/(.*)
> localhost:3142/snapshot.debian.org/archive/\1/${APT_SNAPSHOT_DATE}/\n"
Although, we are using snapshot versions of the packages, we don't use `DISTRO_APT_SNAPSHOT_PREMIRROR` as of now, but will switch soon!
> Then, an apt-cacher-ng instance needs to be started on the host.
> When running inside the kas container, just use the external host IP,
> or forward port 3142 into the container. While the build is running,
> you can monitor the caching statistics under
> http://localhost:3142/acng-report.html.
So for local builds we installed `apt install apt-cacher-ng` and
```
export DISTRO_APT_PREMIRRORS="snapshot-cloudflare.debian.org/archive/debian/(.*) localhost:3142/snapshot-cloudflare.debian.org/archive/debian/20240311/\n \
snapshot-cloudflare.debian.org/archive/debian-security/(.*) localhost:3142/snapshot-cloudflare.debian.org/archive/debian-security/20240311/\n"
```
does the job!
> Second, the cache directory can - in theory - be cached in a CI as
> well, so subsequent jobs can use it. Then, even less load is applied
> onto snapshots.d.o. Even when using standard debian mirrors, this helps
> as usually all ISAR build jobs run in parallel, hence do not share
> already downloaded packages via ISARs internal cache.
For the build server we added a dummy interface to bind on and used something like:
```
image: ghcr.io/siemens/kas/kas-isar:4.0
variables:
APT_CACHE: 192.168.42.1:3142
script:
- export DISTRO_APT_PREMIRRORS="snapshot-cloudflare.debian.org/archive/debian/(.*) ${APT_CACHE}/snapshot-cloudflare.debian.org/archive/debian/20240311/\n snapshot-cloudflare.debian.org/archive/debian-security/(.*) ${APT_CACHE}/snapshot-cloudflare.debian.org/archive/debian-security/20240311/\n"
- kas build ${TARGET}
```
in our CI pipeline.
> Currently I'm playing around with adding a built-in apt-cacher-ng into
> kas and the kas-container. By that, all users can instantly profit from
> the speedup. For CI systems, an ambient caching service (e.g. via
> gitlabs "services") would probably make more sense, but is also more
> complicated to configure.
Usage with kas-container script (from cip-core) did not work for me.
Since the handling of `DISTRO_APT_PREMIRRORS` as well as `--runtime-args`
leads to `docker: invalid reference format.` once spaces are contained,
or, if escaped, the parser used in `isar-bootstrap.inc` could not split it correctly.
(We didn't dig too deep into that, since most of us use kas natively.)
For completeness: We are using:
isar-cip-core @ 95e952569b59602dd05f4138de9602a5b0398f0a
isar @ 3d863aad5e12c7bef382cb6fb3f1f5e18b236156
> I'm happy to hear your thoughts.
Hope that helps :-)
BR Alexander
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/a8e8e8d0-1fa3-4062-a7de-4783808bbc87n%40googlegroups.com.