From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Wed, 13 Nov 2024 15:17:21 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-wr1-f64.google.com (mail-wr1-f64.google.com [209.85.221.64]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 4ADEHL4r028007 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 13 Nov 2024 15:17:21 +0100 Received: by mail-wr1-f64.google.com with SMTP id ffacd0b85a97d-37d458087c0sf503688f8f.1 for ; Wed, 13 Nov 2024 06:17:21 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1731507435; cv=pass; d=google.com; s=arc-20240605; b=SHEiBUkOnzCx8ySVkfcAm+DDnNyMnp8gwyBdiqCvraoxwKtA0ohoR6hlitm2AEelhJ cWOyUP4w0Kzug2u3f+kqEOZWpzYe8tc+vobwC8IxxJMeSD/IXfJ8wevdF+Ll1NLnVRQS 80k9fO+bP6bV+lf9bHggdgkyT+vfQKYZVClPYvjWBOuFTd2hZhleUijrIAKAbp2I+wGa mU6I4WZ8IFGldHYu24NkVUt5F/OFEuZmAnChamZtlC2eokOyaJMgdY8zOEi695A3l4++ YhrFWjXm4U5JnGBD26Z1TSVDogt19TjL8XVL6jKnDIRUFJ02ltOOFoKSA5zZ622sNWMQ sV3Q== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:user-agent :content-transfer-encoding:autocrypt:references:in-reply-to:date:to :from:subject:message-id:sender:dkim-signature; bh=Rb/tBSH1l2MJAzdhWHwt0ATAYDUaWOGVwaup7jqqnHk=; fh=m/TEaCV6nMpG8WcDTBExI84AYBUtzOWnaDVm0UxfAOI=; b=X35oLd8uoHoToHCPwQAnjF7PJVDNzN6VWRzBJnSr3Oom5sFKA2lhtWuFg5En546FyX PFDut3aKMI/kMiaxI8VVH8j3oUug6/t4k44CXg5N39O5HNe52QRAqqs60JbxeInwWnJb l80k0QJBaLtVa+C2MtFWm4Yjzs20JDmHBbpqVRf+U46bWjpnvifcGuNuzCFawSr4M1ge pdh9q1SgbZUoqsaT8TKlAo3S34GMGTxoljZTbWLBOqcLj84uws2FhiSnOqXRvE0gNWUv Jo1yX/iEGgYXpHX4+Hh5Io7ncedRszgdnTxr+EG0hcOvTLBSVmSVH666DM6VEkq6tCE6 z6aQ==; darn=ilbers.de ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1731507435; x=1732112235; darn=ilbers.de; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:user-agent:content-transfer-encoding :autocrypt:references:in-reply-to:date:to:from:subject:message-id :sender:from:to:cc:subject:date:message-id:reply-to; bh=Rb/tBSH1l2MJAzdhWHwt0ATAYDUaWOGVwaup7jqqnHk=; b=jzBdC+EXZXHoS2s5n1CHoGs3/cY8WHwREy3ifVsibhkcDkuuXxUjstY2Q422UuG/O9 Wl0VdPwksJMeJ04DEO+Gx/oahwo4r/1H66o9AtOuVQqYlqhxs/K4RscpERfve4uZon24 B8xHCzeEkmQqpBF3DQGdXL78EGYnoJSpnIEK41IN+VIVbQ8z4Xzr6otbCjT5oJAAo08B docT9wZBYDsDmdb3h0bqamdcS8xnILKFQHzdSPby4DHCmr05hQcolM3UprIDkX7PDx1r z7pe2H+OOAvdS0b+QuODGhJhxbLltrSPwkhLfCbJIDxvrOJ2r+WTZQoL1YieHDqmis/y XKmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731507435; x=1732112235; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :user-agent:content-transfer-encoding:autocrypt:references :in-reply-to:date:to:from:subject:message-id:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=Rb/tBSH1l2MJAzdhWHwt0ATAYDUaWOGVwaup7jqqnHk=; b=W3LRYVoDMW5ynx8hre7wUlaqk2O5uR55wqjIpgev2zUGGWqpxVoUgzn7FCprEYthAx WgI8fhHJcyAaMpsXzfI2Pi74Uih1P85NEm/Zvgw8C2Mim9ifKtKMUu8RU6yGtK76m43w PcTq27Epub5ffmZXwQazGEvsFUTz5D+be7DcVCQZoRqrbcPARhX7VIjjeOvuodyl29Bf rwNNOrTc/qFNNnYEGzcVXjxUc21CnKC37ZXvlnFDUxRhI734e4ZrIwadEzhePh/a2JBq b7bYA0FpebCpc4sP6avYrGHDGI8bLc6osT6BEnAXHLmSpBqdd4kok63OZymCnjhaf+e2 umTA== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCWGWRFWLnNU0gsPZ2L/HNgcB+CRdkHauMfY0LYbYTctarzsGF/8En/4uOLEIG0mf1jcZMkU@ilbers.de X-Gm-Message-State: AOJu0YyMeW3zoNOnB/edpKiB8alhFM1kUX29vLH/uvh32wurmExYL2P3 KDE+PIqzxS4uZ3SE6tQnvdix0xG1awImhqmSILmZnmHRBBdAvgHo X-Google-Smtp-Source: AGHT+IF59QMDZu7eKkqPGxEzDSIgyqe91kGz3Nik0QgD/2i+mx8CesBCga4F6zBpFfxpG1AIrItfAA== X-Received: by 2002:a5d:584b:0:b0:374:b6e4:16a7 with SMTP id ffacd0b85a97d-381f0f583a2mr19814561f8f.8.1731507434686; Wed, 13 Nov 2024 06:17:14 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6000:c8b:b0:371:72ab:2b8e with SMTP id ffacd0b85a97d-3820df6b8f2ls277029f8f.2.-pod-prod-00-eu; Wed, 13 Nov 2024 06:17:11 -0800 (PST) X-Forwarded-Encrypted: i=2; AJvYcCUsXbBNkK34pH8A9pMC0V3y47uK6LeCxw9hRanBeMJj+LaS9GGGlqAmqLkg5iGC7vhAaMBUcdfueaqu@googlegroups.com X-Received: by 2002:a5d:5e0b:0:b0:37d:3b31:7a9d with SMTP id ffacd0b85a97d-381f0f7f221mr13762865f8f.23.1731507431560; Wed, 13 Nov 2024 06:17:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1731507431; cv=none; d=google.com; s=arc-20240605; b=J4tJcs5ti/HlzvKEoyVOwfnoRR4lHlXpT2+5vD77GV9ynGBEDJR+Wc4461DVZjclBE L2oH4k22GEM5qCRMp2Jx2JR7h95L3Rd9+ZbqYMNPddS3rEr1fyReG0NXPz9BcTTXVJz5 GkF+VZpVCSMqy9xJsYCLsiAX4QvQ/ro6bTOkfFXocT0Zr2p62oFnKEDpbEmo/WmYWLfX LRPzw6NeSL7quVUYyDvSA2bcjVCXWNCjNaviqg0smzu8Vm1BelzSwcMe1zjA/0SFJ4UU HuxVslXtPAbmkCS6vsJPeVNeMv8Fw26w4kq8eA+OECoJiXYD/RBcMVc00VKH2JsDoPhK jwkQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=mime-version:user-agent:content-transfer-encoding:autocrypt :references:in-reply-to:date:to:from:subject:message-id; bh=tgcxAUhOqaDNGi9JiVqS4C0EZm/KNbcrYyR7jXXLyT0=; fh=ez+yYIN2+t1/A1KtVD1essDQ6+anTTnZTbSXjuK9WaU=; b=fynaNHTACSU22EgEzbAbwhPg5kuBSp6f2LCbyxhpqpAnr/K3E+YgyEUTHaprW8ucQL chjh3OKXc4d9sb6gyzhjVdbRQZPDbPNW7ak4UK5QmSllLXDOg8st9c4bddcsk9d/wQcY JGEEPRfUcr7lp2ovoYXxbTfJtG/0vLYB5osUqXT+xijXibAm981QFRqtISVrmIcRW9qs 0HWmkg570a6N1fbCUhs9W84Lp0IdGt6qxG8oyjXClugXPKBgmYjVVO3CJxnhTT5SRTZu JA35ro7wvfvntSTOl85jdk6ryyPGYY+QltYpfZQ+2obMRiEynVtBrYPR/3Dhs6S/uH6L 1rpQ==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-432d54fba29si356555e9.2.2024.11.13.06.17.11 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Wed, 13 Nov 2024 06:17:11 -0800 (PST) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from [IPv6:::1] (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 4ADEH96t027996 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Nov 2024 15:17:10 +0100 Message-ID: <0c1bad48830270e1dcb86437830467c7aec9f016.camel@ilbers.de> Subject: Re: [PATCH 1/1] improve apt fetching from rate-limited snapshot mirrors From: Uladzimir Bely To: Felix Moessbauer , isar-users@googlegroups.com Date: Wed, 13 Nov 2024 17:17:09 +0300 In-Reply-To: <20241106124108.220323-1-felix.moessbauer@siemens.com> References: <20241106124108.220323-1-felix.moessbauer@siemens.com> Autocrypt: addr=ubely@ilbers.de; prefer-encrypt=mutual; keydata=mQENBGO2eUkBCACtT+T3OrPVSExBmqfgXT3lp9XcdxRzjYp26wezkgYjjBXaf36bxtaAf S471VoQtpar0RVeFfW7WDDdfX9ZclSj36zBQe+RVSJzoNoNQfjOXWuSHb5Z+cpAFtqBY4muxK4+ia IlLJd6CN3ejOsLHATtCeHHq8wi0z2T+KdLQO+wQRgo2hjj0Lp9pGTrKJry50HP/o7Vbdu14dOx2xq r8+wPc6SQbBIrcqaa4MqCQC00vQG7eXvo+k2MOw59FDdpMH0KR9mHgp3u/s4I+4YRBArukt9G9xz/ rsEFmxAIBC6N/a6Hzwg4puc91n7ABDsPg8Vp+X3MDraujN0dvR6OKVNtABEBAAG0IFVsYWR6aW1pc iBCZWx5IDx1YmVseUBpbGJlcnMuZGU+iQFOBBMBCAA4FiEEJqPNVhVGyk12Eh+PAUQYBM/2FkoFAm O2eUkCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQAUQYBM/2FkprlAgAmuna8Hm9EyoEtTl DBGDg6Zm4ZLp5ffvZBE946h92jepDrteoxsJ7pSzJVC2HmDLa4iZUao7lLLbDsUj5x45/iLJcqBZK k3YnAxP2r6a+kI+1VVQY1pxdG1nlJAbdNzoojm/qmezNPSrqni61KVMQKsXBCWhIjSXDSM9CsBj21 a+9qaVqfxovJGTn9lgrZO+xzKQNMKZeOouJlscVuFj21P0ww3/YENiU/nMeTSuYypO76mDtAd08Jo nc3yuHa9MJGei5ixN3wT+IrGR2aL2hdw2M6NgH7sYbL2Zi4ugD6RXHJai1Bh2yvFSVqSQ+M6QOInT 4ud7wslm1XRB065dXtA== Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.54.1 (by Flathub.org) MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2, RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-TUID: vHnGQkwZlniX On Wed, 2024-11-06 at 13:41 +0100, 'Felix Moessbauer' via isar-users wrote: > The snapshot mirrors apply strict rate limiting, making it impossible > to > use them with a default apt configuration. On the other hand, apt > does > not obey the retry-after http response header, but uses a exponential > backoff mechanism (retry after 1,2,4,...) seconds. By that, the > currently used 3 retries are basically useless as apt aborts way > before > the rate-limit expires. >=20 > We change that to retry at most 10 times with a futher upper limit of > 10 > minutes (both only when using ISAR_USE_APT_SNAPSHOT). This is a > compromise between not stalling failed builds but also making them > more > reliable. >=20 > Signed-off-by: Felix Moessbauer > --- > =C2=A0doc/user_manual.md=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 | 4 +++- > =C2=A0meta/classes/rootfs.bbclass | 5 ++++- > =C2=A0meta/conf/bitbake.conf=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 | 2 ++ > =C2=A03 files changed, 9 insertions(+), 2 deletions(-) >=20 > diff --git a/doc/user_manual.md b/doc/user_manual.md > index 284847c8..4b07f903 100644 > --- a/doc/user_manual.md > +++ b/doc/user_manual.md > @@ -433,7 +433,9 @@ Some other variables include: > =C2=A0 - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to > DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt > bootstrap repo for the host. > =C2=A0 - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the > default URI in the format `ftp.debian.org=C2=A0my.preferred.mirror`. This > variable is optional. PREMIRRORS will be used only for the build. The > final images will have the sources list as mentioned in > DISTRO_APT_SOURCES. > =C2=A0 - `ISAR_USE_APT_SNAPSHOT` - Use a frozen apt snapshot instead of > the live mirror. Optional. > -=C2=A0=C2=A0 `ISAR_APT_DL_LIMIT` - Rate limit the apt fetching to n kB /= s. > Optional. > + - `ISAR_APT_DL_LIMIT` - Rate limit the apt fetching to n kB / s. > Optional. > + - `ISAR_APT_RETRIES` - Number of apt fetching retries before giving > up. Optional > + - `ISAR_APT_DELAY_MAX` - Maximum time in seconds apt performs > retries. Optional > =C2=A0 - `DISTRO_APT_SNAPSHOT_PREMIRROR` - Similar to > `DISTRO_APT_PREMIRRORS` but for a snapshot, pre-defined for supported > distros. > =C2=A0 - `ISAR_APT_SNAPSHOT_TIMESTAMP` - Unix timestamp of the apt > snapshot. Automatically derived from `SOURCE_DATE_EPOCH` if not > overwritten. (Consider `ISAR_APT_SNAPSHOT_DATE` for a more user > friendly format) > =C2=A0 - `ISAR_APT_SNAPSHOT_DATE` - Timestamp in upstream format (e.g. > `20240702T082400Z`) of the apt snapshot. Overrides > `ISAR_APT_SNAPSHOT_TIMESTAMP` if set. Otherwise, will be > automatically derived from `ISAR_APT_SNAPSHOT_TIMESTAMP` > diff --git a/meta/classes/rootfs.bbclass > b/meta/classes/rootfs.bbclass > index c7011508..03e80018 100644 > --- a/meta/classes/rootfs.bbclass > +++ b/meta/classes/rootfs.bbclass > @@ -148,7 +148,10 @@ rootfs_configure_apt() { > =C2=A0 > =C2=A0=C2=A0=C2=A0=C2=A0 mkdir -p '${ROOTFSDIR}/etc/apt/apt.conf.d' > =C2=A0=C2=A0=C2=A0=C2=A0 { > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 echo 'Acquire::Retries "3";' > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 echo 'Acquire::Retries "${ISA= R_APT_RETRIES}";' > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if [ -n "${ISAR_APT_DELAY_MAX= }" ]; then > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 echo = 'Acquire::Retries::Delay::Maximum > "${ISAR_APT_DELAY_MAX}";' > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 fi > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if [ -n "${ISAR_APT_DL_L= IMIT}" ]; then > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = echo 'Acquire::http::Dl-Limit "${ISAR_APT_DL_LIMIT}";' > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = echo 'Acquire::https::Dl-Limit "${ISAR_APT_DL_LIMIT}";' > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > index 13966a62..5fe50787 100644 > --- a/meta/conf/bitbake.conf > +++ b/meta/conf/bitbake.conf > @@ -148,6 +148,8 @@ export SOURCE_DATE_EPOCH ?=3D > "${@get_source_date_epoch_value(d)}" > =C2=A0SOURCE_DATE_EPOCH_FALLBACK ??=3D "1709565251" > =C2=A0# Debian snapshots > =C2=A0ISAR_USE_APT_SNAPSHOT ??=3D "0" > +ISAR_APT_RETRIES ??=3D "${@'10' if > bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else '3'}" > +ISAR_APT_DELAY_MAX ??=3D "${@'600' if > bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else ''}" > =C2=A0ISAR_APT_SNAPSHOT_TIMESTAMP ??=3D "${SOURCE_DATE_EPOCH}" > =C2=A0 > =C2=A0# Default parallelism and resource usage for xz > --=20 > 2.39.5 >=20 Applied to next, thanks. --=20 Best regards, Uladzimir. --=20 You received this message because you are subscribed to the Google Groups "= isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an e= mail to isar-users+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/isar-users/= 0c1bad48830270e1dcb86437830467c7aec9f016.camel%40ilbers.de.