[PATCH 1/1] improve apt fetching from rate-limited snapshot mirrors

[PATCH 1/1] improve apt fetching from rate-limited snapshot mirrors

['Felix Moessbauer' via isar-users]

The snapshot mirrors apply strict rate limiting, making it impossible to
use them with a default apt configuration. On the other hand, apt does
not obey the retry-after http response header, but uses a exponential
backoff mechanism (retry after 1,2,4,...) seconds. By that, the
currently used 3 retries are basically useless as apt aborts way before
the rate-limit expires.

We change that to retry at most 10 times with a futher upper limit of 10
minutes (both only when using ISAR_USE_APT_SNAPSHOT). This is a
compromise between not stalling failed builds but also making them more
reliable.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 doc/user_manual.md          | 4 +++-
 meta/classes/rootfs.bbclass | 5 ++++-
 meta/conf/bitbake.conf      | 2 ++
 3 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 284847c8..4b07f903 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -433,7 +433,9 @@ Some other variables include:
  - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt bootstrap repo for the host.
  - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the default URI in the format `ftp.debian.org my.preferred.mirror`. This variable is optional. PREMIRRORS will be used only for the build. The final images will have the sources list as mentioned in DISTRO_APT_SOURCES.
  - `ISAR_USE_APT_SNAPSHOT` - Use a frozen apt snapshot instead of the live mirror. Optional.
-   `ISAR_APT_DL_LIMIT` - Rate limit the apt fetching to n kB / s. Optional.
+ - `ISAR_APT_DL_LIMIT` - Rate limit the apt fetching to n kB / s. Optional.
+ - `ISAR_APT_RETRIES` - Number of apt fetching retries before giving up. Optional
+ - `ISAR_APT_DELAY_MAX` - Maximum time in seconds apt performs retries. Optional
  - `DISTRO_APT_SNAPSHOT_PREMIRROR` - Similar to `DISTRO_APT_PREMIRRORS` but for a snapshot, pre-defined for supported distros.
  - `ISAR_APT_SNAPSHOT_TIMESTAMP` - Unix timestamp of the apt snapshot. Automatically derived from `SOURCE_DATE_EPOCH` if not overwritten. (Consider `ISAR_APT_SNAPSHOT_DATE` for a more user friendly format)
  - `ISAR_APT_SNAPSHOT_DATE` - Timestamp in upstream format (e.g. `20240702T082400Z`) of the apt snapshot. Overrides `ISAR_APT_SNAPSHOT_TIMESTAMP` if set. Otherwise, will be automatically derived from `ISAR_APT_SNAPSHOT_TIMESTAMP`
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index c7011508..03e80018 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -148,7 +148,10 @@ rootfs_configure_apt() {
 
     mkdir -p '${ROOTFSDIR}/etc/apt/apt.conf.d'
     {
-        echo 'Acquire::Retries "3";'
+        echo 'Acquire::Retries "${ISAR_APT_RETRIES}";'
+        if [ -n "${ISAR_APT_DELAY_MAX}" ]; then
+            echo 'Acquire::Retries::Delay::Maximum "${ISAR_APT_DELAY_MAX}";'
+        fi
         if [ -n "${ISAR_APT_DL_LIMIT}" ]; then
             echo 'Acquire::http::Dl-Limit "${ISAR_APT_DL_LIMIT}";'
             echo 'Acquire::https::Dl-Limit "${ISAR_APT_DL_LIMIT}";'
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 13966a62..5fe50787 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -148,6 +148,8 @@ export SOURCE_DATE_EPOCH ?= "${@get_source_date_epoch_value(d)}"
 SOURCE_DATE_EPOCH_FALLBACK ??= "1709565251"
 # Debian snapshots
 ISAR_USE_APT_SNAPSHOT ??= "0"
+ISAR_APT_RETRIES ??= "${@'10' if bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else '3'}"
+ISAR_APT_DELAY_MAX ??= "${@'600' if bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else ''}"
 ISAR_APT_SNAPSHOT_TIMESTAMP ??= "${SOURCE_DATE_EPOCH}"
 
 # Default parallelism and resource usage for xz
-- 
2.39.5

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20241106124108.220323-1-felix.moessbauer%40siemens.com.

Re: [PATCH 1/1] improve apt fetching from rate-limited snapshot mirrors

['Jan Kiszka' via isar-users]

On 06.11.24 13:41, Felix Moessbauer wrote:
> The snapshot mirrors apply strict rate limiting, making it impossible to
> use them with a default apt configuration. On the other hand, apt does
> not obey the retry-after http response header, but uses a exponential
> backoff mechanism (retry after 1,2,4,...) seconds. By that, the
> currently used 3 retries are basically useless as apt aborts way before
> the rate-limit expires.
> 
> We change that to retry at most 10 times with a futher upper limit of 10
> minutes (both only when using ISAR_USE_APT_SNAPSHOT). This is a
> compromise between not stalling failed builds but also making them more
> reliable.
> 
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
>  doc/user_manual.md          | 4 +++-
>  meta/classes/rootfs.bbclass | 5 ++++-
>  meta/conf/bitbake.conf      | 2 ++
>  3 files changed, 9 insertions(+), 2 deletions(-)
> 
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 284847c8..4b07f903 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -433,7 +433,9 @@ Some other variables include:
>   - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt bootstrap repo for the host.
>   - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the default URI in the format `ftp.debian.org my.preferred.mirror`. This variable is optional. PREMIRRORS will be used only for the build. The final images will have the sources list as mentioned in DISTRO_APT_SOURCES.
>   - `ISAR_USE_APT_SNAPSHOT` - Use a frozen apt snapshot instead of the live mirror. Optional.
> -   `ISAR_APT_DL_LIMIT` - Rate limit the apt fetching to n kB / s. Optional.
> + - `ISAR_APT_DL_LIMIT` - Rate limit the apt fetching to n kB / s. Optional.
> + - `ISAR_APT_RETRIES` - Number of apt fetching retries before giving up. Optional
> + - `ISAR_APT_DELAY_MAX` - Maximum time in seconds apt performs retries. Optional
>   - `DISTRO_APT_SNAPSHOT_PREMIRROR` - Similar to `DISTRO_APT_PREMIRRORS` but for a snapshot, pre-defined for supported distros.
>   - `ISAR_APT_SNAPSHOT_TIMESTAMP` - Unix timestamp of the apt snapshot. Automatically derived from `SOURCE_DATE_EPOCH` if not overwritten. (Consider `ISAR_APT_SNAPSHOT_DATE` for a more user friendly format)
>   - `ISAR_APT_SNAPSHOT_DATE` - Timestamp in upstream format (e.g. `20240702T082400Z`) of the apt snapshot. Overrides `ISAR_APT_SNAPSHOT_TIMESTAMP` if set. Otherwise, will be automatically derived from `ISAR_APT_SNAPSHOT_TIMESTAMP`
> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
> index c7011508..03e80018 100644
> --- a/meta/classes/rootfs.bbclass
> +++ b/meta/classes/rootfs.bbclass
> @@ -148,7 +148,10 @@ rootfs_configure_apt() {
>  
>      mkdir -p '${ROOTFSDIR}/etc/apt/apt.conf.d'
>      {
> -        echo 'Acquire::Retries "3";'
> +        echo 'Acquire::Retries "${ISAR_APT_RETRIES}";'
> +        if [ -n "${ISAR_APT_DELAY_MAX}" ]; then
> +            echo 'Acquire::Retries::Delay::Maximum "${ISAR_APT_DELAY_MAX}";'
> +        fi
>          if [ -n "${ISAR_APT_DL_LIMIT}" ]; then
>              echo 'Acquire::http::Dl-Limit "${ISAR_APT_DL_LIMIT}";'
>              echo 'Acquire::https::Dl-Limit "${ISAR_APT_DL_LIMIT}";'
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 13966a62..5fe50787 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -148,6 +148,8 @@ export SOURCE_DATE_EPOCH ?= "${@get_source_date_epoch_value(d)}"
>  SOURCE_DATE_EPOCH_FALLBACK ??= "1709565251"
>  # Debian snapshots
>  ISAR_USE_APT_SNAPSHOT ??= "0"
> +ISAR_APT_RETRIES ??= "${@'10' if bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else '3'}"
> +ISAR_APT_DELAY_MAX ??= "${@'600' if bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else ''}"
>  ISAR_APT_SNAPSHOT_TIMESTAMP ??= "${SOURCE_DATE_EPOCH}"
>  
>  # Default parallelism and resource usage for xz

Seems to help, just unbroke the riscv build of isar-cip-core with that
which works against snapshot.d.o so far.

jan

-- 
Siemens AG, Technology
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/a284d67e-e779-4d42-b942-57ab6843bd46%40siemens.com.

Re: [PATCH 1/1] improve apt fetching from rate-limited snapshot mirrors

['MOESSBAUER, Felix' via isar-users]

On Wed, 2024-11-06 at 13:41 +0100, Felix Moessbauer wrote:
> The snapshot mirrors apply strict rate limiting, making it impossible
> to
> use them with a default apt configuration. 

Hi, can we please merge this soon-ish, if there are no objections? We
already have multiple downstream layers that carry this patch to be
able to build against snapshots again.

At the same time I'm working on improve the performance of snapshot.d.o
, together with the upstream maintainers.

Best regards,
Felix

> On the other hand, apt does
> not obey the retry-after http response header, but uses a exponential
> backoff mechanism (retry after 1,2,4,...) seconds. By that, the
> currently used 3 retries are basically useless as apt aborts way
> before
> the rate-limit expires.
> 
> We change that to retry at most 10 times with a futher upper limit of
> 10
> minutes (both only when using ISAR_USE_APT_SNAPSHOT). This is a
> compromise between not stalling failed builds but also making them
> more
> reliable.
> 
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
>  doc/user_manual.md          | 4 +++-
>  meta/classes/rootfs.bbclass | 5 ++++-
>  meta/conf/bitbake.conf      | 2 ++
>  3 files changed, 9 insertions(+), 2 deletions(-)
> 
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 284847c8..4b07f903 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -433,7 +433,9 @@ Some other variables include:
>   - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to
> DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt
> bootstrap repo for the host.
>   - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the
> default URI in the format `ftp.debian.org my.preferred.mirror`. This
> variable is optional. PREMIRRORS will be used only for the build. The
> final images will have the sources list as mentioned in
> DISTRO_APT_SOURCES.
>   - `ISAR_USE_APT_SNAPSHOT` - Use a frozen apt snapshot instead of
> the live mirror. Optional.
> -   `ISAR_APT_DL_LIMIT` - Rate limit the apt fetching to n kB / s.
> Optional.
> + - `ISAR_APT_DL_LIMIT` - Rate limit the apt fetching to n kB / s.
> Optional.
> + - `ISAR_APT_RETRIES` - Number of apt fetching retries before giving
> up. Optional
> + - `ISAR_APT_DELAY_MAX` - Maximum time in seconds apt performs
> retries. Optional
>   - `DISTRO_APT_SNAPSHOT_PREMIRROR` - Similar to
> `DISTRO_APT_PREMIRRORS` but for a snapshot, pre-defined for supported
> distros.
>   - `ISAR_APT_SNAPSHOT_TIMESTAMP` - Unix timestamp of the apt
> snapshot. Automatically derived from `SOURCE_DATE_EPOCH` if not
> overwritten. (Consider `ISAR_APT_SNAPSHOT_DATE` for a more user
> friendly format)
>   - `ISAR_APT_SNAPSHOT_DATE` - Timestamp in upstream format (e.g.
> `20240702T082400Z`) of the apt snapshot. Overrides
> `ISAR_APT_SNAPSHOT_TIMESTAMP` if set. Otherwise, will be
> automatically derived from `ISAR_APT_SNAPSHOT_TIMESTAMP`
> diff --git a/meta/classes/rootfs.bbclass
> b/meta/classes/rootfs.bbclass
> index c7011508..03e80018 100644
> --- a/meta/classes/rootfs.bbclass
> +++ b/meta/classes/rootfs.bbclass
> @@ -148,7 +148,10 @@ rootfs_configure_apt() {
>  
>      mkdir -p '${ROOTFSDIR}/etc/apt/apt.conf.d'
>      {
> -        echo 'Acquire::Retries "3";'
> +        echo 'Acquire::Retries "${ISAR_APT_RETRIES}";'
> +        if [ -n "${ISAR_APT_DELAY_MAX}" ]; then
> +            echo 'Acquire::Retries::Delay::Maximum
> "${ISAR_APT_DELAY_MAX}";'
> +        fi
>          if [ -n "${ISAR_APT_DL_LIMIT}" ]; then
>              echo 'Acquire::http::Dl-Limit "${ISAR_APT_DL_LIMIT}";'
>              echo 'Acquire::https::Dl-Limit "${ISAR_APT_DL_LIMIT}";'
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 13966a62..5fe50787 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -148,6 +148,8 @@ export SOURCE_DATE_EPOCH ?=
> "${@get_source_date_epoch_value(d)}"
>  SOURCE_DATE_EPOCH_FALLBACK ??= "1709565251"
>  # Debian snapshots
>  ISAR_USE_APT_SNAPSHOT ??= "0"
> +ISAR_APT_RETRIES ??= "${@'10' if
> bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else '3'}"
> +ISAR_APT_DELAY_MAX ??= "${@'600' if
> bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else ''}"
>  ISAR_APT_SNAPSHOT_TIMESTAMP ??= "${SOURCE_DATE_EPOCH}"
>  
>  # Default parallelism and resource usage for xz

-- 
Siemens AG, Technology
Linux Expert Center


-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/c71f9988e2b8a394930188726be1830dc480b35d.camel%40siemens.com.

Re: [PATCH 1/1] improve apt fetching from rate-limited snapshot mirrors

[Uladzimir Bely]

On Wed, 2024-11-06 at 13:41 +0100, 'Felix Moessbauer' via isar-users
wrote:
> The snapshot mirrors apply strict rate limiting, making it impossible
> to
> use them with a default apt configuration. On the other hand, apt
> does
> not obey the retry-after http response header, but uses a exponential
> backoff mechanism (retry after 1,2,4,...) seconds. By that, the
> currently used 3 retries are basically useless as apt aborts way
> before
> the rate-limit expires.
> 
> We change that to retry at most 10 times with a futher upper limit of
> 10
> minutes (both only when using ISAR_USE_APT_SNAPSHOT). This is a
> compromise between not stalling failed builds but also making them
> more
> reliable.
> 
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
>  doc/user_manual.md          | 4 +++-
>  meta/classes/rootfs.bbclass | 5 ++++-
>  meta/conf/bitbake.conf      | 2 ++
>  3 files changed, 9 insertions(+), 2 deletions(-)
> 
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 284847c8..4b07f903 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -433,7 +433,9 @@ Some other variables include:
>   - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to
> DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt
> bootstrap repo for the host.
>   - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the
> default URI in the format `ftp.debian.org my.preferred.mirror`. This
> variable is optional. PREMIRRORS will be used only for the build. The
> final images will have the sources list as mentioned in
> DISTRO_APT_SOURCES.
>   - `ISAR_USE_APT_SNAPSHOT` - Use a frozen apt snapshot instead of
> the live mirror. Optional.
> -   `ISAR_APT_DL_LIMIT` - Rate limit the apt fetching to n kB / s.
> Optional.
> + - `ISAR_APT_DL_LIMIT` - Rate limit the apt fetching to n kB / s.
> Optional.
> + - `ISAR_APT_RETRIES` - Number of apt fetching retries before giving
> up. Optional
> + - `ISAR_APT_DELAY_MAX` - Maximum time in seconds apt performs
> retries. Optional
>   - `DISTRO_APT_SNAPSHOT_PREMIRROR` - Similar to
> `DISTRO_APT_PREMIRRORS` but for a snapshot, pre-defined for supported
> distros.
>   - `ISAR_APT_SNAPSHOT_TIMESTAMP` - Unix timestamp of the apt
> snapshot. Automatically derived from `SOURCE_DATE_EPOCH` if not
> overwritten. (Consider `ISAR_APT_SNAPSHOT_DATE` for a more user
> friendly format)
>   - `ISAR_APT_SNAPSHOT_DATE` - Timestamp in upstream format (e.g.
> `20240702T082400Z`) of the apt snapshot. Overrides
> `ISAR_APT_SNAPSHOT_TIMESTAMP` if set. Otherwise, will be
> automatically derived from `ISAR_APT_SNAPSHOT_TIMESTAMP`
> diff --git a/meta/classes/rootfs.bbclass
> b/meta/classes/rootfs.bbclass
> index c7011508..03e80018 100644
> --- a/meta/classes/rootfs.bbclass
> +++ b/meta/classes/rootfs.bbclass
> @@ -148,7 +148,10 @@ rootfs_configure_apt() {
>  
>      mkdir -p '${ROOTFSDIR}/etc/apt/apt.conf.d'
>      {
> -        echo 'Acquire::Retries "3";'
> +        echo 'Acquire::Retries "${ISAR_APT_RETRIES}";'
> +        if [ -n "${ISAR_APT_DELAY_MAX}" ]; then
> +            echo 'Acquire::Retries::Delay::Maximum
> "${ISAR_APT_DELAY_MAX}";'
> +        fi
>          if [ -n "${ISAR_APT_DL_LIMIT}" ]; then
>              echo 'Acquire::http::Dl-Limit "${ISAR_APT_DL_LIMIT}";'
>              echo 'Acquire::https::Dl-Limit "${ISAR_APT_DL_LIMIT}";'
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 13966a62..5fe50787 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -148,6 +148,8 @@ export SOURCE_DATE_EPOCH ?=
> "${@get_source_date_epoch_value(d)}"
>  SOURCE_DATE_EPOCH_FALLBACK ??= "1709565251"
>  # Debian snapshots
>  ISAR_USE_APT_SNAPSHOT ??= "0"
> +ISAR_APT_RETRIES ??= "${@'10' if
> bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else '3'}"
> +ISAR_APT_DELAY_MAX ??= "${@'600' if
> bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else ''}"
>  ISAR_APT_SNAPSHOT_TIMESTAMP ??= "${SOURCE_DATE_EPOCH}"
>  
>  # Default parallelism and resource usage for xz
> -- 
> 2.39.5
> 

Applied to next, thanks.

-- 
Best regards,
Uladzimir.



-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/0c1bad48830270e1dcb86437830467c7aec9f016.camel%40ilbers.de.