Re: [PATCH 1/1] improve apt fetching from rate-limited snapshot mirrors

["'MOESSBAUER, Felix' via isar-users" <isar-users@googlegroups.com>]

On Wed, 2024-11-06 at 13:41 +0100, Felix Moessbauer wrote:
> The snapshot mirrors apply strict rate limiting, making it impossible
> to
> use them with a default apt configuration. 

Hi, can we please merge this soon-ish, if there are no objections? We
already have multiple downstream layers that carry this patch to be
able to build against snapshots again.

At the same time I'm working on improve the performance of snapshot.d.o
, together with the upstream maintainers.

Best regards,
Felix

> On the other hand, apt does
> not obey the retry-after http response header, but uses a exponential
> backoff mechanism (retry after 1,2,4,...) seconds. By that, the
> currently used 3 retries are basically useless as apt aborts way
> before
> the rate-limit expires.
> 
> We change that to retry at most 10 times with a futher upper limit of
> 10
> minutes (both only when using ISAR_USE_APT_SNAPSHOT). This is a
> compromise between not stalling failed builds but also making them
> more
> reliable.
> 
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
>  doc/user_manual.md          | 4 +++-
>  meta/classes/rootfs.bbclass | 5 ++++-
>  meta/conf/bitbake.conf      | 2 ++
>  3 files changed, 9 insertions(+), 2 deletions(-)
> 
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 284847c8..4b07f903 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -433,7 +433,9 @@ Some other variables include:
>   - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to
> DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt
> bootstrap repo for the host.
>   - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the
> default URI in the format `ftp.debian.org my.preferred.mirror`. This
> variable is optional. PREMIRRORS will be used only for the build. The
> final images will have the sources list as mentioned in
> DISTRO_APT_SOURCES.
>   - `ISAR_USE_APT_SNAPSHOT` - Use a frozen apt snapshot instead of
> the live mirror. Optional.
> -   `ISAR_APT_DL_LIMIT` - Rate limit the apt fetching to n kB / s.
> Optional.
> + - `ISAR_APT_DL_LIMIT` - Rate limit the apt fetching to n kB / s.
> Optional.
> + - `ISAR_APT_RETRIES` - Number of apt fetching retries before giving
> up. Optional
> + - `ISAR_APT_DELAY_MAX` - Maximum time in seconds apt performs
> retries. Optional
>   - `DISTRO_APT_SNAPSHOT_PREMIRROR` - Similar to
> `DISTRO_APT_PREMIRRORS` but for a snapshot, pre-defined for supported
> distros.
>   - `ISAR_APT_SNAPSHOT_TIMESTAMP` - Unix timestamp of the apt
> snapshot. Automatically derived from `SOURCE_DATE_EPOCH` if not
> overwritten. (Consider `ISAR_APT_SNAPSHOT_DATE` for a more user
> friendly format)
>   - `ISAR_APT_SNAPSHOT_DATE` - Timestamp in upstream format (e.g.
> `20240702T082400Z`) of the apt snapshot. Overrides
> `ISAR_APT_SNAPSHOT_TIMESTAMP` if set. Otherwise, will be
> automatically derived from `ISAR_APT_SNAPSHOT_TIMESTAMP`
> diff --git a/meta/classes/rootfs.bbclass
> b/meta/classes/rootfs.bbclass
> index c7011508..03e80018 100644
> --- a/meta/classes/rootfs.bbclass
> +++ b/meta/classes/rootfs.bbclass
> @@ -148,7 +148,10 @@ rootfs_configure_apt() {
>  
>      mkdir -p '${ROOTFSDIR}/etc/apt/apt.conf.d'
>      {
> -        echo 'Acquire::Retries "3";'
> +        echo 'Acquire::Retries "${ISAR_APT_RETRIES}";'
> +        if [ -n "${ISAR_APT_DELAY_MAX}" ]; then
> +            echo 'Acquire::Retries::Delay::Maximum
> "${ISAR_APT_DELAY_MAX}";'
> +        fi
>          if [ -n "${ISAR_APT_DL_LIMIT}" ]; then
>              echo 'Acquire::http::Dl-Limit "${ISAR_APT_DL_LIMIT}";'
>              echo 'Acquire::https::Dl-Limit "${ISAR_APT_DL_LIMIT}";'
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 13966a62..5fe50787 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -148,6 +148,8 @@ export SOURCE_DATE_EPOCH ?=
> "${@get_source_date_epoch_value(d)}"
>  SOURCE_DATE_EPOCH_FALLBACK ??= "1709565251"
>  # Debian snapshots
>  ISAR_USE_APT_SNAPSHOT ??= "0"
> +ISAR_APT_RETRIES ??= "${@'10' if
> bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else '3'}"
> +ISAR_APT_DELAY_MAX ??= "${@'600' if
> bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else ''}"
>  ISAR_APT_SNAPSHOT_TIMESTAMP ??= "${SOURCE_DATE_EPOCH}"
>  
>  # Default parallelism and resource usage for xz

-- 
Siemens AG, Technology
Linux Expert Center


-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/c71f9988e2b8a394930188726be1830dc480b35d.camel%40siemens.com.