Re: [RFC][PATCH] rootfs: Retry downloads 3 times

[Henning Schild <henning.schild@siemens.com>]

Am Fri, 8 May 2020 07:57:34 +0200
schrieb Jan Kiszka <jan.kiszka@siemens.com>:

> On 07.05.20 20:13, Henning Schild wrote:
> > The idea is good but the patch is incomplete.
> > 
> > To find all places where we download you might want to look at
> > bitbake fetchers in general - do they have a retry count?
> > 
> > Is there a variable controlling that?  
> 
> As explained in the commit log, it's not consistent in bitbake. At
> least wget has such a retry counter. In other cases, bitbake seems to
> quickly fall back to mirrors and fail.

Ok now i know how to read that.
 
> > 
> > After that look at all debian downloaders. You will find them when
> > looking for "download-only", "apt-get update" any maybe more. They
> > should use the central bitbake count or switch if available.
> > 
> > Your patch found one of many "download-only" and ignores bitbake.
> > Might be the most likely ... but we should try to be consistent.  
> 
> Yes, we likely also need this when fetching build deps. All images
> are covered this way already, I also checked debootstrap but found no
> retry switch there. So the result will remain "best effort" in
> general. Therefore, I was also wondering if it's worth it.

I do not think so. We do have the cache now, maybe we can
deb_dl_dir_export after failed "download-only". That way you can at
least retry on the outside. So interactive incremental builds should
work on a shaky connection.

In a later step we could introduce the arg for apt and write a loop
around debian fetchers that can not retry ... debootstrap. That could
help reduce the number of manual increments, and maybe help
non-interactive builds with shaky networks.

Henning

> Background was may lengthy attempt yesterday to install a complex
> image from snapshot.debian.org. The "solution" (that avoided a first
> complete local mirror) was to throttle the download bandwidth to some
> 1 MBit/s. Maybe there was just one bad mirror on the other side, but
> it constantly killed the image installed. Retrying was a futile
> attempt there, but I've also seen (few) nightly builds in the past
> that stumbled over individual packet downloads where this might have
> helped.
>
> Jan
> 
> > 
> > Henning
> > 
> > Am Thu, 7 May 2020 16:36:18 +0200
> > schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> >   
> >> From: Jan Kiszka <jan.kiszka@siemens.com>
> >>
> >> Avoids failing a complete rootfs installation in case of a short
> >> hick-up.
> >>
> >> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> >> ---
> >>
> >> Does something like this make sense? Do we have more places?
> >> We have a  retry of 2 on wget bitbake fetches, e.g. (via
> >> FETCHCMD_wget).
> >>
> >>   meta/classes/rootfs.bbclass | 2 +-
> >>   1 file changed, 1 insertion(+), 1 deletion(-)
> >>
> >> diff --git a/meta/classes/rootfs.bbclass
> >> b/meta/classes/rootfs.bbclass index 806e824c..afec1cbc 100644
> >> --- a/meta/classes/rootfs.bbclass
> >> +++ b/meta/classes/rootfs.bbclass
> >> @@ -123,7 +123,7 @@ rootfs_install_pkgs_download[weight] = "600"
> >>   rootfs_install_pkgs_download[isar-apt-lock] = "release-after"
> >>   rootfs_install_pkgs_download() {
> >>       sudo -E chroot '${ROOTFSDIR}' \
> >> -        /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only
> >> ${ROOTFS_PACKAGES}
> >> +        /usr/bin/apt-get ${ROOTFS_APT_ARGS} -o Acquire::Retries=3
> >> --download-only ${ROOTFS_PACKAGES} }
> >>   
> >>   ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT ??= ""  
> >   
> 
>