Re: [RFC][PATCH] rootfs: Retry downloads 3 times

[Jan Kiszka <jan.kiszka@siemens.com>]

On 07.05.20 20:13, Henning Schild wrote:
> The idea is good but the patch is incomplete.
> 
> To find all places where we download you might want to look at bitbake
> fetchers in general - do they have a retry count?
> 
> Is there a variable controlling that?

As explained in the commit log, it's not consistent in bitbake. At least 
wget has such a retry counter. In other cases, bitbake seems to quickly 
fall back to mirrors and fail.

> 
> After that look at all debian downloaders. You will find them when
> looking for "download-only", "apt-get update" any maybe more. They
> should use the central bitbake count or switch if available.
> 
> Your patch found one of many "download-only" and ignores bitbake. Might
> be the most likely ... but we should try to be consistent.

Yes, we likely also need this when fetching build deps. All images are 
covered this way already, I also checked debootstrap but found no retry 
switch there. So the result will remain "best effort" in general. 
Therefore, I was also wondering if it's worth it.

Background was may lengthy attempt yesterday to install a complex image 
from snapshot.debian.org. The "solution" (that avoided a first complete 
local mirror) was to throttle the download bandwidth to some 1 MBit/s. 
Maybe there was just one bad mirror on the other side, but it constantly 
killed the image installed. Retrying was a futile attempt there, but 
I've also seen (few) nightly builds in the past that stumbled over 
individual packet downloads where this might have helped.

Jan

> 
> Henning
> 
> Am Thu, 7 May 2020 16:36:18 +0200
> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> 
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> Avoids failing a complete rootfs installation in case of a short
>> hick-up.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>
>> Does something like this make sense? Do we have more places?
>> We have a  retry of 2 on wget bitbake fetches, e.g. (via
>> FETCHCMD_wget).
>>
>>   meta/classes/rootfs.bbclass | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
>> index 806e824c..afec1cbc 100644
>> --- a/meta/classes/rootfs.bbclass
>> +++ b/meta/classes/rootfs.bbclass
>> @@ -123,7 +123,7 @@ rootfs_install_pkgs_download[weight] = "600"
>>   rootfs_install_pkgs_download[isar-apt-lock] = "release-after"
>>   rootfs_install_pkgs_download() {
>>       sudo -E chroot '${ROOTFSDIR}' \
>> -        /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only
>> ${ROOTFS_PACKAGES}
>> +        /usr/bin/apt-get ${ROOTFS_APT_ARGS} -o Acquire::Retries=3
>> --download-only ${ROOTFS_PACKAGES} }
>>   
>>   ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT ??= ""
> 


-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux