[RFC][PATCH] rootfs: Retry downloads 3 times

[RFC][PATCH] rootfs: Retry downloads 3 times

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Avoids failing a complete rootfs installation in case of a short
hick-up.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---

Does something like this make sense? Do we have more places?
We have a  retry of 2 on wget bitbake fetches, e.g. (via FETCHCMD_wget).

 meta/classes/rootfs.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 806e824c..afec1cbc 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -123,7 +123,7 @@ rootfs_install_pkgs_download[weight] = "600"
 rootfs_install_pkgs_download[isar-apt-lock] = "release-after"
 rootfs_install_pkgs_download() {
     sudo -E chroot '${ROOTFSDIR}' \
-        /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only ${ROOTFS_PACKAGES}
+        /usr/bin/apt-get ${ROOTFS_APT_ARGS} -o Acquire::Retries=3 --download-only ${ROOTFS_PACKAGES}
 }
 
 ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT ??= ""
-- 
2.26.1

Re: [RFC][PATCH] rootfs: Retry downloads 3 times

[Henning Schild]

The idea is good but the patch is incomplete.

To find all places where we download you might want to look at bitbake
fetchers in general - do they have a retry count?

Is there a variable controlling that?

After that look at all debian downloaders. You will find them when
looking for "download-only", "apt-get update" any maybe more. They
should use the central bitbake count or switch if available.

Your patch found one of many "download-only" and ignores bitbake. Might
be the most likely ... but we should try to be consistent.

Henning

Am Thu, 7 May 2020 16:36:18 +0200
schrieb Jan Kiszka <jan.kiszka@siemens.com>:

> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> Avoids failing a complete rootfs installation in case of a short
> hick-up.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
> 
> Does something like this make sense? Do we have more places?
> We have a  retry of 2 on wget bitbake fetches, e.g. (via
> FETCHCMD_wget).
> 
>  meta/classes/rootfs.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
> index 806e824c..afec1cbc 100644
> --- a/meta/classes/rootfs.bbclass
> +++ b/meta/classes/rootfs.bbclass
> @@ -123,7 +123,7 @@ rootfs_install_pkgs_download[weight] = "600"
>  rootfs_install_pkgs_download[isar-apt-lock] = "release-after"
>  rootfs_install_pkgs_download() {
>      sudo -E chroot '${ROOTFSDIR}' \
> -        /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only
> ${ROOTFS_PACKAGES}
> +        /usr/bin/apt-get ${ROOTFS_APT_ARGS} -o Acquire::Retries=3
> --download-only ${ROOTFS_PACKAGES} }
>  
>  ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT ??= ""

Re: [RFC][PATCH] rootfs: Retry downloads 3 times

[Jan Kiszka]

On 07.05.20 20:13, Henning Schild wrote:
> The idea is good but the patch is incomplete.
> 
> To find all places where we download you might want to look at bitbake
> fetchers in general - do they have a retry count?
> 
> Is there a variable controlling that?

As explained in the commit log, it's not consistent in bitbake. At least 
wget has such a retry counter. In other cases, bitbake seems to quickly 
fall back to mirrors and fail.

> 
> After that look at all debian downloaders. You will find them when
> looking for "download-only", "apt-get update" any maybe more. They
> should use the central bitbake count or switch if available.
> 
> Your patch found one of many "download-only" and ignores bitbake. Might
> be the most likely ... but we should try to be consistent.

Yes, we likely also need this when fetching build deps. All images are 
covered this way already, I also checked debootstrap but found no retry 
switch there. So the result will remain "best effort" in general. 
Therefore, I was also wondering if it's worth it.

Background was may lengthy attempt yesterday to install a complex image 
from snapshot.debian.org. The "solution" (that avoided a first complete 
local mirror) was to throttle the download bandwidth to some 1 MBit/s. 
Maybe there was just one bad mirror on the other side, but it constantly 
killed the image installed. Retrying was a futile attempt there, but 
I've also seen (few) nightly builds in the past that stumbled over 
individual packet downloads where this might have helped.

Jan

> 
> Henning
> 
> Am Thu, 7 May 2020 16:36:18 +0200
> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> 
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> Avoids failing a complete rootfs installation in case of a short
>> hick-up.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>
>> Does something like this make sense? Do we have more places?
>> We have a  retry of 2 on wget bitbake fetches, e.g. (via
>> FETCHCMD_wget).
>>
>>   meta/classes/rootfs.bbclass | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
>> index 806e824c..afec1cbc 100644
>> --- a/meta/classes/rootfs.bbclass
>> +++ b/meta/classes/rootfs.bbclass
>> @@ -123,7 +123,7 @@ rootfs_install_pkgs_download[weight] = "600"
>>   rootfs_install_pkgs_download[isar-apt-lock] = "release-after"
>>   rootfs_install_pkgs_download() {
>>       sudo -E chroot '${ROOTFSDIR}' \
>> -        /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only
>> ${ROOTFS_PACKAGES}
>> +        /usr/bin/apt-get ${ROOTFS_APT_ARGS} -o Acquire::Retries=3
>> --download-only ${ROOTFS_PACKAGES} }
>>   
>>   ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT ??= ""
> 


-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [RFC][PATCH] rootfs: Retry downloads 3 times

[Henning Schild]

Am Fri, 8 May 2020 07:57:34 +0200
schrieb Jan Kiszka <jan.kiszka@siemens.com>:

> On 07.05.20 20:13, Henning Schild wrote:
> > The idea is good but the patch is incomplete.
> > 
> > To find all places where we download you might want to look at
> > bitbake fetchers in general - do they have a retry count?
> > 
> > Is there a variable controlling that?  
> 
> As explained in the commit log, it's not consistent in bitbake. At
> least wget has such a retry counter. In other cases, bitbake seems to
> quickly fall back to mirrors and fail.

Ok now i know how to read that.
 
> > 
> > After that look at all debian downloaders. You will find them when
> > looking for "download-only", "apt-get update" any maybe more. They
> > should use the central bitbake count or switch if available.
> > 
> > Your patch found one of many "download-only" and ignores bitbake.
> > Might be the most likely ... but we should try to be consistent.  
> 
> Yes, we likely also need this when fetching build deps. All images
> are covered this way already, I also checked debootstrap but found no
> retry switch there. So the result will remain "best effort" in
> general. Therefore, I was also wondering if it's worth it.

I do not think so. We do have the cache now, maybe we can
deb_dl_dir_export after failed "download-only". That way you can at
least retry on the outside. So interactive incremental builds should
work on a shaky connection.

In a later step we could introduce the arg for apt and write a loop
around debian fetchers that can not retry ... debootstrap. That could
help reduce the number of manual increments, and maybe help
non-interactive builds with shaky networks.

Henning

> Background was may lengthy attempt yesterday to install a complex
> image from snapshot.debian.org. The "solution" (that avoided a first
> complete local mirror) was to throttle the download bandwidth to some
> 1 MBit/s. Maybe there was just one bad mirror on the other side, but
> it constantly killed the image installed. Retrying was a futile
> attempt there, but I've also seen (few) nightly builds in the past
> that stumbled over individual packet downloads where this might have
> helped.
>
> Jan
> 
> > 
> > Henning
> > 
> > Am Thu, 7 May 2020 16:36:18 +0200
> > schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> >   
> >> From: Jan Kiszka <jan.kiszka@siemens.com>
> >>
> >> Avoids failing a complete rootfs installation in case of a short
> >> hick-up.
> >>
> >> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> >> ---
> >>
> >> Does something like this make sense? Do we have more places?
> >> We have a  retry of 2 on wget bitbake fetches, e.g. (via
> >> FETCHCMD_wget).
> >>
> >>   meta/classes/rootfs.bbclass | 2 +-
> >>   1 file changed, 1 insertion(+), 1 deletion(-)
> >>
> >> diff --git a/meta/classes/rootfs.bbclass
> >> b/meta/classes/rootfs.bbclass index 806e824c..afec1cbc 100644
> >> --- a/meta/classes/rootfs.bbclass
> >> +++ b/meta/classes/rootfs.bbclass
> >> @@ -123,7 +123,7 @@ rootfs_install_pkgs_download[weight] = "600"
> >>   rootfs_install_pkgs_download[isar-apt-lock] = "release-after"
> >>   rootfs_install_pkgs_download() {
> >>       sudo -E chroot '${ROOTFSDIR}' \
> >> -        /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only
> >> ${ROOTFS_PACKAGES}
> >> +        /usr/bin/apt-get ${ROOTFS_APT_ARGS} -o Acquire::Retries=3
> >> --download-only ${ROOTFS_PACKAGES} }
> >>   
> >>   ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT ??= ""  
> >   
> 
> 

Re: [RFC][PATCH] rootfs: Retry downloads 3 times

[Baurzhan Ismagulov]

On Thu, May 07, 2020 at 04:36:18PM +0200, Jan Kiszka wrote:
> Avoids failing a complete rootfs installation in case of a short
> hick-up.
...
> Does something like this make sense? Do we have more places?
> We have a  retry of 2 on wget bitbake fetches, e.g. (via FETCHCMD_wget).

To address the questions raised in the thread, I think a small improvement now
is a good thing. I also think it's worth to gradually fix all relevant places
and unify the retry handling.

I'd like to apply this as is. Jan, you are not planning an update for this, are
you?

With kind regards,
Baurzhan.

Re: [RFC][PATCH] rootfs: Retry downloads 3 times

[Jan Kiszka]

On 25.05.20 16:10, Baurzhan Ismagulov wrote:
> On Thu, May 07, 2020 at 04:36:18PM +0200, Jan Kiszka wrote:
>> Avoids failing a complete rootfs installation in case of a short
>> hick-up.
> ...
>> Does something like this make sense? Do we have more places?
>> We have a  retry of 2 on wget bitbake fetches, e.g. (via FETCHCMD_wget).
> 
> To address the questions raised in the thread, I think a small improvement now
> is a good thing. I also think it's worth to gradually fix all relevant places
> and unify the retry handling.
> 
> I'd like to apply this as is. Jan, you are not planning an update for this, are
> you?
> 

No update planned. If you like to take it, I'm fine, if not, I'm also fine.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [RFC][PATCH] rootfs: Retry downloads 3 times

[Baurzhan Ismagulov]

On Mon, May 25, 2020 at 04:13:37PM +0200, Jan Kiszka wrote:
> No update planned.

Applied to next, thanks.

With kind regards,
Baurzhan.