Re: [PATCH 1/1] improve apt fetching from rate-limited snapshot mirrors

[Uladzimir Bely <ubely@ilbers.de>]

On Wed, 2024-11-06 at 13:41 +0100, 'Felix Moessbauer' via isar-users
wrote:
> The snapshot mirrors apply strict rate limiting, making it impossible
> to
> use them with a default apt configuration. On the other hand, apt
> does
> not obey the retry-after http response header, but uses a exponential
> backoff mechanism (retry after 1,2,4,...) seconds. By that, the
> currently used 3 retries are basically useless as apt aborts way
> before
> the rate-limit expires.
> 
> We change that to retry at most 10 times with a futher upper limit of
> 10
> minutes (both only when using ISAR_USE_APT_SNAPSHOT). This is a
> compromise between not stalling failed builds but also making them
> more
> reliable.
> 
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
>  doc/user_manual.md          | 4 +++-
>  meta/classes/rootfs.bbclass | 5 ++++-
>  meta/conf/bitbake.conf      | 2 ++
>  3 files changed, 9 insertions(+), 2 deletions(-)
> 
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 284847c8..4b07f903 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -433,7 +433,9 @@ Some other variables include:
>   - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to
> DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt
> bootstrap repo for the host.
>   - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the
> default URI in the format `ftp.debian.org my.preferred.mirror`. This
> variable is optional. PREMIRRORS will be used only for the build. The
> final images will have the sources list as mentioned in
> DISTRO_APT_SOURCES.
>   - `ISAR_USE_APT_SNAPSHOT` - Use a frozen apt snapshot instead of
> the live mirror. Optional.
> -   `ISAR_APT_DL_LIMIT` - Rate limit the apt fetching to n kB / s.
> Optional.
> + - `ISAR_APT_DL_LIMIT` - Rate limit the apt fetching to n kB / s.
> Optional.
> + - `ISAR_APT_RETRIES` - Number of apt fetching retries before giving
> up. Optional
> + - `ISAR_APT_DELAY_MAX` - Maximum time in seconds apt performs
> retries. Optional
>   - `DISTRO_APT_SNAPSHOT_PREMIRROR` - Similar to
> `DISTRO_APT_PREMIRRORS` but for a snapshot, pre-defined for supported
> distros.
>   - `ISAR_APT_SNAPSHOT_TIMESTAMP` - Unix timestamp of the apt
> snapshot. Automatically derived from `SOURCE_DATE_EPOCH` if not
> overwritten. (Consider `ISAR_APT_SNAPSHOT_DATE` for a more user
> friendly format)
>   - `ISAR_APT_SNAPSHOT_DATE` - Timestamp in upstream format (e.g.
> `20240702T082400Z`) of the apt snapshot. Overrides
> `ISAR_APT_SNAPSHOT_TIMESTAMP` if set. Otherwise, will be
> automatically derived from `ISAR_APT_SNAPSHOT_TIMESTAMP`
> diff --git a/meta/classes/rootfs.bbclass
> b/meta/classes/rootfs.bbclass
> index c7011508..03e80018 100644
> --- a/meta/classes/rootfs.bbclass
> +++ b/meta/classes/rootfs.bbclass
> @@ -148,7 +148,10 @@ rootfs_configure_apt() {
>  
>      mkdir -p '${ROOTFSDIR}/etc/apt/apt.conf.d'
>      {
> -        echo 'Acquire::Retries "3";'
> +        echo 'Acquire::Retries "${ISAR_APT_RETRIES}";'
> +        if [ -n "${ISAR_APT_DELAY_MAX}" ]; then
> +            echo 'Acquire::Retries::Delay::Maximum
> "${ISAR_APT_DELAY_MAX}";'
> +        fi
>          if [ -n "${ISAR_APT_DL_LIMIT}" ]; then
>              echo 'Acquire::http::Dl-Limit "${ISAR_APT_DL_LIMIT}";'
>              echo 'Acquire::https::Dl-Limit "${ISAR_APT_DL_LIMIT}";'
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 13966a62..5fe50787 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -148,6 +148,8 @@ export SOURCE_DATE_EPOCH ?=
> "${@get_source_date_epoch_value(d)}"
>  SOURCE_DATE_EPOCH_FALLBACK ??= "1709565251"
>  # Debian snapshots
>  ISAR_USE_APT_SNAPSHOT ??= "0"
> +ISAR_APT_RETRIES ??= "${@'10' if
> bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else '3'}"
> +ISAR_APT_DELAY_MAX ??= "${@'600' if
> bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else ''}"
>  ISAR_APT_SNAPSHOT_TIMESTAMP ??= "${SOURCE_DATE_EPOCH}"
>  
>  # Default parallelism and resource usage for xz
> -- 
> 2.39.5
> 

Applied to next, thanks.

-- 
Best regards,
Uladzimir.



-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/0c1bad48830270e1dcb86437830467c7aec9f016.camel%40ilbers.de.