From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 22 Aug 2024 13:10:41 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-oa1-f57.google.com (mail-oa1-f57.google.com [209.85.160.57]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 47MBAdnB030366 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 22 Aug 2024 13:10:40 +0200 Received: by mail-oa1-f57.google.com with SMTP id 586e51a60fabf-2701355d099sf714507fac.0 for ; Thu, 22 Aug 2024 04:10:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1724325034; x=1724929834; darn=ilbers.de; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:x-original-sender :mime-version:subject:references:in-reply-to:message-id:to:from:date :from:to:cc:subject:date:message-id:reply-to; bh=3fFyi4mzWRNC1wzlsmLlrqaEisPaWqXaiJI1wdg0yW8=; b=frhONd9GK/9+Xd8L4hrkgBvPLEYGvKYFfiawELSfKMnRWwWspYPcpmZZnnZ6gFbMfX 0NcEApQr9bfwEuVTfm2n2dOOCTjXqmo711ulauuFbkx6lXHwLREBNSfAPcVmHxPlkJ9m gOxHM7qrk4P4kYnhsIHR8uRzHOOuXWB6TPco5eDxirCwsyMnszkvYSGZiInd2gGxMRp6 1XNMMUKOijiNv+hpF3RVYAlUaioAQNHTlrqC6nvOlLeiKtF4uCR8nBVSZSjMMs5I9/ed zllqdBlW3azGm6v6hptUNmNvECoUGXAHNQb6FfEgT9lfoALoOli1VDS/MCgGSWu8YxP7 zYRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724325034; x=1724929834; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-sender:mime-version:subject:references:in-reply-to :message-id:to:from:date:x-beenthere:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3fFyi4mzWRNC1wzlsmLlrqaEisPaWqXaiJI1wdg0yW8=; b=SoQFNrDzvXhd/kIGXdTxnb2V01gElOxlfdVEn1skZVYm1XBJJdaGmNw/xr0gteSVrJ 5n00iNPUK9Gz3gU+Efa3yBBfZxWid3hmv+UO4K7UHT34ahpvB0/SMiVoKgt3lWbF4mH7 ytX2iYeMxushNLzuRGVpTUX46+H+ZZO8/wspKlBtJJwYOiaSw3Xx+0QXSuubnoaBO0cZ qoJnyeAlEnMkAyXRVOrE8wEyYsEi1H23NeWI6ksiBHB86HEGjWAkgMPQK1a+4OapeIyS ZNVZQ60Pn9ao7nxkGDKrnUsIz3/HKe2WM0mXAfj+N6EmSVk4sGiOAtdOZiWZ8hznsEcv HwMg== X-Forwarded-Encrypted: i=1; AJvYcCWUPNuhmZ4AXizli2yWcFonXXiw0fQZuyNoIeJamKL8gK8GEQj3RC1RjsrU+4jYuGjQx+FZ@ilbers.de X-Gm-Message-State: AOJu0YzBdGHVhHyA0Muq0Mu2E08Z2zf/CSBauufTIVG9lm4+jiR8GdgB lylKhNtPP8OI8X4d7VLu10nT0DHOP7wveJKULDAbQgx9BXzNQSwu X-Google-Smtp-Source: AGHT+IFiP7o9QguzdtmK9eFdUeqAw0eGHS9FqcB7lFPliBMadvcMm3zpQyjCKcfNv1SZol0NIM5OBQ== X-Received: by 2002:a05:6870:3041:b0:26c:5312:a0f9 with SMTP id 586e51a60fabf-2738be5ad29mr6163221fac.33.1724325033919; Thu, 22 Aug 2024 04:10:33 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6870:bb0d:b0:25d:f93b:9be0 with SMTP id 586e51a60fabf-273ca949e23ls1007708fac.1.-pod-prod-05-us; Thu, 22 Aug 2024 04:10:33 -0700 (PDT) X-Received: by 2002:a05:6358:5e09:b0:1ac:ef39:fd6 with SMTP id e5c5f4694b2df-1b5b3b0a9a3mr182300955d.28.1724325032737; Thu, 22 Aug 2024 04:10:32 -0700 (PDT) Date: Thu, 22 Aug 2024 04:10:32 -0700 (PDT) From: "'Alexander Heinisch' via isar-users" To: isar-users Message-Id: In-Reply-To: <18f0728819aa994072fc6c9ff0eda58dffc6b6b6.camel@siemens.com> References: <18f0728819aa994072fc6c9ff0eda58dffc6b6b6.camel@siemens.com> Subject: Re: Reliability and build time improvements with apt-cacher-ng MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_34298_78157926.1724325032524" X-Original-Sender: alexander.heinisch@siemens.com X-Original-From: Alexander Heinisch Reply-To: Alexander Heinisch Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,HTML_MESSAGE,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: aJf2H/gCdZxM ------=_Part_34298_78157926.1724325032524 Content-Type: multipart/alternative; boundary="----=_Part_34299_1482228072.1724325032524" ------=_Part_34299_1482228072.1724325032524 Content-Type: text/plain; charset="UTF-8" Hi Felix! > I just played around with apt-cacher-ng to circumvent the throttling > issues on snapshots.d.o, and got some impressive results: We just migrated our gitlab-runners (for a downstream project based on isar-cip-core) to a new build server. The project builds 4 images (all for amd64 arch) whereas 2 of them are done in a multiconfig setup using the isar-installer. In total the complete build fetches 423MB of apt packages from snapshot-cloudflare.debian.org >From an initial build time of ~12min (~35min on the old build server :-)) we could reduce build times (building all 4 images) down to ~7min using apt-cacher-ng. > First of all, it is easy to configure ISAR to use the apt-cacher-ng, by > just using the DISTRO_APT_PREMIRRORS: Yes, we just passed `DISTRO_APT_PREMIRRORS` to our kas yaml like: ``` env: DISTRO_APT_PREMIRRORS: "" ``` > DISTRO_APT_SNAPSHOT_PREMIRROR = "deb.debian.org/(.*) > localhost:3142/snapshot.debian.org/archive/\1/${APT_SNAPSHOT_DATE}/\n" Although, we are using snapshot versions of the packages, we don't use `DISTRO_APT_SNAPSHOT_PREMIRROR` as of now, but will switch soon! > Then, an apt-cacher-ng instance needs to be started on the host. > When running inside the kas container, just use the external host IP, > or forward port 3142 into the container. While the build is running, > you can monitor the caching statistics under > http://localhost:3142/acng-report.html. So for local builds we installed `apt install apt-cacher-ng` and ``` export DISTRO_APT_PREMIRRORS="snapshot-cloudflare.debian.org/archive/debian/(.*) localhost:3142/snapshot-cloudflare.debian.org/archive/debian/20240311/\n \ snapshot-cloudflare.debian.org/archive/debian-security/(.*) localhost:3142/snapshot-cloudflare.debian.org/archive/debian-security/20240311/\n" ``` does the job! > Second, the cache directory can - in theory - be cached in a CI as > well, so subsequent jobs can use it. Then, even less load is applied > onto snapshots.d.o. Even when using standard debian mirrors, this helps > as usually all ISAR build jobs run in parallel, hence do not share > already downloaded packages via ISARs internal cache. For the build server we added a dummy interface to bind on and used something like: ``` image: ghcr.io/siemens/kas/kas-isar:4.0 variables: APT_CACHE: 192.168.42.1:3142 script: - export DISTRO_APT_PREMIRRORS="snapshot-cloudflare.debian.org/archive/debian/(.*) ${APT_CACHE}/snapshot-cloudflare.debian.org/archive/debian/20240311/\n snapshot-cloudflare.debian.org/archive/debian-security/(.*) ${APT_CACHE}/snapshot-cloudflare.debian.org/archive/debian-security/20240311/\n" - kas build ${TARGET} ``` in our CI pipeline. > Currently I'm playing around with adding a built-in apt-cacher-ng into > kas and the kas-container. By that, all users can instantly profit from > the speedup. For CI systems, an ambient caching service (e.g. via > gitlabs "services") would probably make more sense, but is also more > complicated to configure. Usage with kas-container script (from cip-core) did not work for me. Since the handling of `DISTRO_APT_PREMIRRORS` as well as `--runtime-args` leads to `docker: invalid reference format.` once spaces are contained, or, if escaped, the parser used in `isar-bootstrap.inc` could not split it correctly. (We didn't dig too deep into that, since most of us use kas natively.) For completeness: We are using: isar-cip-core @ 95e952569b59602dd05f4138de9602a5b0398f0a isar @ 3d863aad5e12c7bef382cb6fb3f1f5e18b236156 > I'm happy to hear your thoughts. Hope that helps :-) BR Alexander -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/a8e8e8d0-1fa3-4062-a7de-4783808bbc87n%40googlegroups.com. ------=_Part_34299_1482228072.1724325032524 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Felix!

> I just played around with apt-cacher-ng to circum= vent the throttling
> issues on snapshots.d.o, and got some impress= ive results:

We just migrated our gitlab-runners (for a downstre= am project based on isar-cip-core) to a new build server.
The project = builds 4 images (all for amd64 arch) whereas 2 of them are done in a multic= onfig setup using the isar-installer.
In total the complete build fetc= hes 423MB of apt packages from snapshot-cloudflare.debian.org

Fr= om an initial build time of ~12min (~35min on the old build server :-)) we = could reduce build times (building all 4 images) down to ~7min using apt-ca= cher-ng.

> First of all, it is easy to configure ISAR to use = the apt-cacher-ng, by
> just using the DISTRO_APT_PREMIRRORS:
=
Yes, we just passed `DISTRO_APT_PREMIRRORS` to our kas yaml like:

```
env:
=C2=A0 DISTRO_APT_PREMIRRORS: ""
```
<= br />> DISTRO_APT_SNAPSHOT_PREMIRROR =3D "deb.debian.org/(.*)
> = localhost:3142/snapshot.debian.org/archive/\1/${APT_SNAPSHOT_DATE}/\n"

Although, we are using snapshot versions of the packages, we don't u= se=C2=A0`DISTRO_APT_SNAPSHOT_PREMIRROR`=C2=A0 as of now, but will switch so= on!
=C2=A0
> Then, an apt-cacher-ng instance needs to be start= ed on the host.
> When running inside the kas container, just use t= he external host IP,
> or forward port 3142 into the container. Whi= le the build is running,
> you can monitor the caching statistics u= nder
> http://localhost:3142/acng-report.html.
=C2=A0
So = for local builds we installed `apt install apt-cacher-ng`=C2=A0and
```=
export DISTRO_APT_PREMIRRORS=3D"snapshot-cloudflare.debian.org/archiv= e/debian/(.*) localhost:3142/snapshot-cloudflare.debian.org/archive/debian/= 20240311/\n \
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 snapshot-cloudflare.debian= .org/archive/debian-security/(.*) localhost:3142/snapshot-cloudflare.debian= .org/archive/debian-security/20240311/\n"
```
does the job!
=
> Second, the cache directory can - in theory - be cached in a CI = as
> well, so subsequent jobs can use it. Then, even less load is a= pplied
> onto snapshots.d.o. Even when using standard debian mirror= s, this helps
> as usually all ISAR build jobs run in parallel, hen= ce do not share
> already downloaded packages via ISARs internal ca= che.

For the build server we added a dummy interface to bind on = and used something like:

```
image: ghcr.io/siemens/kas/kas= -isar:4.0
variables:
=C2=A0 APT_CACHE: 192.168.42.1:3142
scr= ipt:
=C2=A0 - export DISTRO_APT_PREMIRRORS=3D"snapshot-cloudflare.debi= an.org/archive/debian/(.*) ${APT_CACHE}/snapshot-cloudflare.debian.org/arch= ive/debian/20240311/\n snapshot-cloudflare.debian.org/archive/debian-securi= ty/(.*) ${APT_CACHE}/snapshot-cloudflare.debian.org/archive/debian-security= /20240311/\n"
=C2=A0 - kas build ${TARGET}
```
in our CI pip= eline.
=C2=A0
> Currently I'm playing around with adding a bui= lt-in apt-cacher-ng into
> kas and the kas-container. By that, all = users can instantly profit from
> the speedup. For CI systems, an a= mbient caching service (e.g. via
> gitlabs "services") would probab= ly make more sense, but is also more
> complicated to configure.
Usage with kas-container script (from cip-core) did not work for m= e.
Since the handling of `DISTRO_APT_PREMIRRORS` as well as `--runtime= -args`
leads to `docker: invalid reference format.` once spaces are c= ontained,
or, if escaped, the parser used in `isar-bootstrap.inc` coul= d not split it correctly.
(We didn't dig too deep into that, since mos= t of us use kas natively.)
For completeness: We are using:
isar-c= ip-core @=C2=A095e952569b59602dd05f4138de9602a5b0398f0a
isar @=C2=A03d= 863aad5e12c7bef382cb6fb3f1f5e18b236156

> I'm happy to hear yo= ur thoughts.

Hope that helps :-)

BR Alexander

--
You received this message because you are subscribed to the Google Groups &= quot;isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an e= mail to isar-use= rs+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msg= id/isar-users/a8e8e8d0-1fa3-4062-a7de-4783808bbc87n%40googlegroups.com.=
------=_Part_34299_1482228072.1724325032524-- ------=_Part_34298_78157926.1724325032524--