From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Wed, 06 Nov 2024 13:41:28 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-ot1-f64.google.com (mail-ot1-f64.google.com [209.85.210.64]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 4A6CfRR3018556 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 6 Nov 2024 13:41:28 +0100 Received: by mail-ot1-f64.google.com with SMTP id 46e09a7af769-71841b0ee34sf607243a34.3 for ; Wed, 06 Nov 2024 04:41:27 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1730896881; cv=pass; d=google.com; s=arc-20240605; b=MsaLw/1fDUXq8jXuivevMWpMXYwIbfMxMZOUnDqvFmGdwrd/JATk008L4Fv/JcW++B rnYpGlKi0etnVRKaX0KITTV9UIC283b1pgXOoctnTS1VUiMoJlH1zF7EXMkxiqnLcTGd zAgoMmlGItlwLfOpLVddM5lFofue5QhhhHamwGi7v3Fg40lW9FuTmRT33a/23cXIRXmX bJ/F0ikZ8JrHt04BnRz7l6UutBLxBxUePaS8Oj6ibM0MG79jCYtlhdWz+hQnGcT1CeK0 tHNZJb1R9CuTx0607tHS2PMx22437iN2N+R/l4reKCwXxNtAdgChmI0ufS3R4x9pwWB/ GteA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id :content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:dkim-signature; bh=YrraIzorot2dg7mHR9EI2VlQvPxihjBZwVD8m04TL8g=; fh=NkfQJymuXZQZ2Hr+zRCts3M13k8lJ7xFe3UcNyWGbeE=; b=INzUN/piefgduzNJMCt89fpon/OA3RNlnIURjZgEt3dwW9kRgFZ32XiwhJ5WbjrJDv 0tDubmXMdDEa30w6UPNX4K3fCJb8/t/TuLaMjUnLKJw8vMCZCHp8lW6amwB6/17Q25Bj UbS4mJ99MWd7rcRbjCPXKr3PaRX2KKXaRVqCkaQS5OelvMLQpGWu28JQtuhbQQ8LgIJk MfnN981fyxUVLdbm4s9f7on5SxqKddRSlyENiaz2JrGcQp3lpsqKGTczdx6J4lhA8ig8 B59KvhlkwWaEMM7LBouK2YjxFE8VzECFKd5SSzb/jKkig1HrDRtC9e5TTetI5z18UYXA 5MJA==; darn=ilbers.de ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b="iIK1YU/u"; spf=pass (google.com: domain of fm-1321639-20241106124117ea6b334caf08935df3-w0doqv@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-1321639-20241106124117ea6b334caf08935df3-w0DoqV@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1730896881; x=1731501681; darn=ilbers.de; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=YrraIzorot2dg7mHR9EI2VlQvPxihjBZwVD8m04TL8g=; b=lTAr/vQYP7SD3kAnW0Q0QT8Qvifyd9GJfsfX1Kt/TxIyFoQbpIxmXOcewh9JNAS/SH 8EH/U5kCbcgS2N6DluWYcyBtRvdUJYlCYHuhV4EdRDoH8UKyCc8Xq8JJ/0Zc2nDDcwEk owXZs/xzter6vLMxDZw7hLuEj7b0uFH7OLJMSLxUk/AdO0+69jqzsIGfb22nQJqbb/WC RG8qZaKuYlUVYa+8PIKPvOautrB0NeZ5RFf02O2McUydFC2Go7ZDdqyFAa33DN/HUd0x 6ySnpOK4kmzIfutLzOcP3EutkBHKMQm0WDM0mBzRbEn96VNudK+wBK+lJhA1EDtafBnC jufA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730896881; x=1731501681; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=YrraIzorot2dg7mHR9EI2VlQvPxihjBZwVD8m04TL8g=; b=axuLDLaKPXmjXbWeIxM5NwldgXBm8z7cK/zKcq5veGjhJ4d/mz6HVhLbrkfe6gMADT al1ZHrb3CFyo9TbopwcEcfUNrLaASVu2DaSNw/ScZvCTcX9L6QNoK/srBKvPaJpnTFCp uJ5fajD8EixsjUvWQ/nikNO7RBFnM/ZlU5Qqp+kq7kFa3ViIpoMmjKQlCIHy3d8Rhyzf 9hG5kjcxHJk7jloFDVtReVgOwoql0w3EiQxeuWCLXU0j3kzpqwDMFLVUqENiGdWB63DS S9lhvo/WJuRGvsX2QjCiWgVUoCNZh8ejwdJZo70WriMzdGgYClLgpbnjSt0xFKbKquxj 6oZQ== X-Forwarded-Encrypted: i=2; AJvYcCXnoVXNHK5KJOzpx9EbXFFGfabbv/VvJ9sVF5AuLRLV0P+kpSHD9nuGXpLW/amo4FcNn69p@ilbers.de X-Gm-Message-State: AOJu0YzVKJ/h/Tq9FfeagAVgb7FIkFXuwIvvIpIKR5F5LG1BFTABeMMP EBukF4fSQuq1U0s6RK36h4kbZYvsREyFFGNHUV0BZ9iluSx+jyvW X-Google-Smtp-Source: AGHT+IGEoRJn2JJr3RpV++pSfMtw8ut3lDpU7sH5MDR/MvdOpPTSZ+7RhhMbUHodzwwXlF83gF8NSg== X-Received: by 2002:a05:6870:7311:b0:288:2955:3efb with SMTP id 586e51a60fabf-29051c4e8a4mr9211589fac.8.1730896881083; Wed, 06 Nov 2024 04:41:21 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6870:82a4:b0:278:2606:8489 with SMTP id 586e51a60fabf-294820f6998ls5718624fac.0.-pod-prod-09-us; Wed, 06 Nov 2024 04:41:20 -0800 (PST) X-Received: by 2002:a05:6870:8a20:b0:261:877:7459 with SMTP id 586e51a60fabf-29051d39578mr35290644fac.38.1730896880126; Wed, 06 Nov 2024 04:41:20 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1730896880; cv=none; d=google.com; s=arc-20240605; b=Un2iu0NMGnEdazVGsuw5QpMoilyEG3YDvsvLHwEHysO9NbdS14f5P4C3rKo658rOqf 37EOP3X2tuzR3wdoOjV/s0/ddoJ+SuZamdQR61L8TDG0JJvun5LEcbYfVuyN95sQAzc9 To0lHnKH5GS8T5E10oKaQQfh6LWlpVjOM6xLYh23OlXUAYpTL+/XYFKn32aZntgm6PQD st7g/moQZEplq8PRWR8ELlfCxtxGD08SdpCPa6GTauX9vY5h4BG43mLrM+IMxHUXmZi0 cTC5zrwLJwv6RJ8vItCtSsYzLetz21wWV3g/hauiKK+GaJ7oN6VEcouuOStNYqyv0Hao 9hzQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=feedback-id:content-transfer-encoding:mime-version:message-id:date :subject:cc:to:from:dkim-signature; bh=E8xuwmHL3B2w9jjXybcrxpjo1xzFOT6FNTdA8pCpAGo=; fh=QHjmU9cI+NAmDDBS7+5Yc1hf2rJyWRDM00VAGqng09w=; b=G34MAQVEhEwBi6xNYoghiMnZzH0rkOqESo2Pybg0E8kShjN9n+/Aw8pC3e4sTf5p1U NTDA73v4/zccys2j3leidafcZpMZyX+kp5griEB386CftBbFuZt97LvothpEERt9pHGa GZBUAY0+rOiru604AvVL8/7PASfICMyb0mRhJYfjaQ+WUGcWwAZMY5CkQRiqOqjzkMpb ERdvyq5qeKfNGp/DhjgRMg46QX2ZR7UEZM9d+GWsunPmocPQeFwf7aVfGcYUapH7a3CD 4ilbCelsD6o/Ujlvgyf6XHv6+FDkvy8HgzGAVE56hLUbcQ25ofMSiEJsTVS4SFQ9Go53 6S3Q==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b="iIK1YU/u"; spf=pass (google.com: domain of fm-1321639-20241106124117ea6b334caf08935df3-w0doqv@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-1321639-20241106124117ea6b334caf08935df3-w0DoqV@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-65-226.siemens.flowmailer.net (mta-65-226.siemens.flowmailer.net. [185.136.65.226]) by gmr-mx.google.com with ESMTPS id 46e09a7af769-7189cc6ca47si587140a34.2.2024.11.06.04.41.19 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 06 Nov 2024 04:41:19 -0800 (PST) Received-SPF: pass (google.com: domain of fm-1321639-20241106124117ea6b334caf08935df3-w0doqv@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) client-ip=185.136.65.226; Received: by mta-65-226.siemens.flowmailer.net with ESMTPSA id 20241106124117ea6b334caf08935df3 for ; Wed, 06 Nov 2024 13:41:18 +0100 From: "'Felix Moessbauer' via isar-users" To: isar-users@googlegroups.com Cc: florian.bezdeka@siemens.com, jan.kiszka@siemens.com, alexander.heinisch@siemens.com, Felix Moessbauer Subject: [PATCH 1/1] improve apt fetching from rate-limited snapshot mirrors Date: Wed, 6 Nov 2024 13:41:08 +0100 Message-Id: <20241106124108.220323-1-felix.moessbauer@siemens.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1321639:519-21489:flowmailer X-Original-Sender: felix.moessbauer@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b="iIK1YU/u"; spf=pass (google.com: domain of fm-1321639-20241106124117ea6b334caf08935df3-w0doqv@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-1321639-20241106124117ea6b334caf08935df3-w0DoqV@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Felix Moessbauer Reply-To: Felix Moessbauer Content-Type: text/plain; charset="UTF-8" Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: gLT7sULJWeBx The snapshot mirrors apply strict rate limiting, making it impossible to use them with a default apt configuration. On the other hand, apt does not obey the retry-after http response header, but uses a exponential backoff mechanism (retry after 1,2,4,...) seconds. By that, the currently used 3 retries are basically useless as apt aborts way before the rate-limit expires. We change that to retry at most 10 times with a futher upper limit of 10 minutes (both only when using ISAR_USE_APT_SNAPSHOT). This is a compromise between not stalling failed builds but also making them more reliable. Signed-off-by: Felix Moessbauer --- doc/user_manual.md | 4 +++- meta/classes/rootfs.bbclass | 5 ++++- meta/conf/bitbake.conf | 2 ++ 3 files changed, 9 insertions(+), 2 deletions(-) diff --git a/doc/user_manual.md b/doc/user_manual.md index 284847c8..4b07f903 100644 --- a/doc/user_manual.md +++ b/doc/user_manual.md @@ -433,7 +433,9 @@ Some other variables include: - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to DISTRO_BOOTSTRAP_KEYS: Li= st of gpg key URIs used to verify apt bootstrap repo for the host. - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the defaul= t URI in the format `ftp.debian.org my.preferred.mirror`. This variable is = optional. PREMIRRORS will be used only for the build. The final images will= have the sources list as mentioned in DISTRO_APT_SOURCES. - `ISAR_USE_APT_SNAPSHOT` - Use a frozen apt snapshot instead of the live= mirror. Optional. - `ISAR_APT_DL_LIMIT` - Rate limit the apt fetching to n kB / s. Optional= . + - `ISAR_APT_DL_LIMIT` - Rate limit the apt fetching to n kB / s. Optional= . + - `ISAR_APT_RETRIES` - Number of apt fetching retries before giving up. O= ptional + - `ISAR_APT_DELAY_MAX` - Maximum time in seconds apt performs retries. Op= tional - `DISTRO_APT_SNAPSHOT_PREMIRROR` - Similar to `DISTRO_APT_PREMIRRORS` bu= t for a snapshot, pre-defined for supported distros. - `ISAR_APT_SNAPSHOT_TIMESTAMP` - Unix timestamp of the apt snapshot. Aut= omatically derived from `SOURCE_DATE_EPOCH` if not overwritten. (Consider `= ISAR_APT_SNAPSHOT_DATE` for a more user friendly format) - `ISAR_APT_SNAPSHOT_DATE` - Timestamp in upstream format (e.g. `20240702= T082400Z`) of the apt snapshot. Overrides `ISAR_APT_SNAPSHOT_TIMESTAMP` if = set. Otherwise, will be automatically derived from `ISAR_APT_SNAPSHOT_TIMES= TAMP` diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index c7011508..03e80018 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -148,7 +148,10 @@ rootfs_configure_apt() { =20 mkdir -p '${ROOTFSDIR}/etc/apt/apt.conf.d' { - echo 'Acquire::Retries "3";' + echo 'Acquire::Retries "${ISAR_APT_RETRIES}";' + if [ -n "${ISAR_APT_DELAY_MAX}" ]; then + echo 'Acquire::Retries::Delay::Maximum "${ISAR_APT_DELAY_MAX}"= ;' + fi if [ -n "${ISAR_APT_DL_LIMIT}" ]; then echo 'Acquire::http::Dl-Limit "${ISAR_APT_DL_LIMIT}";' echo 'Acquire::https::Dl-Limit "${ISAR_APT_DL_LIMIT}";' diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 13966a62..5fe50787 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -148,6 +148,8 @@ export SOURCE_DATE_EPOCH ?=3D "${@get_source_date_epoch= _value(d)}" SOURCE_DATE_EPOCH_FALLBACK ??=3D "1709565251" # Debian snapshots ISAR_USE_APT_SNAPSHOT ??=3D "0" +ISAR_APT_RETRIES ??=3D "${@'10' if bb.utils.to_boolean(d.getVar('ISAR_USE_= APT_SNAPSHOT')) else '3'}" +ISAR_APT_DELAY_MAX ??=3D "${@'600' if bb.utils.to_boolean(d.getVar('ISAR_U= SE_APT_SNAPSHOT')) else ''}" ISAR_APT_SNAPSHOT_TIMESTAMP ??=3D "${SOURCE_DATE_EPOCH}" =20 # Default parallelism and resource usage for xz --=20 2.39.5 --=20 You received this message because you are subscribed to the Google Groups "= isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an e= mail to isar-users+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/isar-users/= 20241106124108.220323-1-felix.moessbauer%40siemens.com.