Re: [PATCH] isar-bootstrap: Increase cache room

[Henning Schild <henning.schild@siemens.com>]

Am Tue, 8 May 2018 10:56:55 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> On 05/07/2018 08:12 PM, Henning Schild wrote:
> > Am Mon, 7 May 2018 19:48:16 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >   
> >> Hi,
> >>
> >> On 05/07/2018 07:31 PM, Henning Schild wrote:  
> >>> Am Mon, 7 May 2018 18:57:36 +0300
> >>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >>>      
> >>>> On 05/07/2018 06:48 PM, Alexander Smirnov wrote:  
> >>>>> According to the man page for 'apt.conf', there are three
> >>>>> variables:
> >>>>>     - Cache-Start: initial size of mmap cache room
> >>>>>     - Cache-Grow: gap for dynamic mmap cache growth
> >>>>>     - Cache-Limit: maximal cache size after growth
> >>>>>
> >>>>> If dynamic resize of mmap region is not avaialbe, the apt uses
> >>>>> pre-allocated Cache-Start room [1] for mmap file.
> >>>>>
> >>>>> Building Isar on one of the Debian host with kernel 3.4, I got
> >>>>> the following problem for 'qemuarm64-stretch' configuration:
> >>>>>
> >>>>> 8<--
> >>>>> Hit:1 http://security.debian.org stretch/updates InRelease
> >>>>> Ign:2 http://ftp.de.debian.org/debian stretch InRelease
> >>>>> Hit:3 http://ftp.de.debian.org/debian stretch-updates InRelease
> >>>>> Hit:4 http://ftp.de.debian.org/debian stretch Release
> >>>>> E: Dynamic MMap ran out of room. Please increase the size of
> >>>>> APT::Cache-Start. Current value: 25165824. (man 5 apt.conf)
> >>>>> qemu: uncaught target signal 11 (Segmentation fault) - core
> >>>>> dumped Segmentation fault 8<--
> >>>>>
> >>>>> I have no information, why exactly the room could not be
> >>>>> re-sized on this system, but it would be better to increase the
> >>>>> initial room size for apt. This patch increases the default apt
> >>>>> cache twice.  
> >>>>
> >>>> 1: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=657085
> >>>>
> >>>> Missed :-(  
> >>>
> >>> What is your conclusion from that link and what does it mean for
> >>> the patch?
> >>>      
> >>
> >> This just the information I found about dynamic cache increasing.
> >> Sometimes it could not work. Another fact is that increasing of
> >> Cache-Limit doesn't help, so seems that mremap really doesn't work.
> >>  
> >>> I do not like adding magic numbers and changing the default
> >>> config, especially if the whole reason is to work around a
> >>> problem only found on outdated build hosts (3.4 really?)  
> >>
> >> I'm not 100% sure that kernel 3.4 is the source of the problem.
> >> Tried to investigate, but for now without results.
> >>
> >> But if you try to google line:
> >>
> >>     E: Dynamic MMap ran out of room. Please increase the size of
> >> APT::Cache-Start
> >>
> >> There will be tons of search results, so such problem occurred not
> >> only on my host.
> >>
> >>
> >> The man page says:
> >>
> >> ...on systems with a lot of configured sources it should be
> >> increased...
> >>
> >> so for me it makes sense to increase this room even if there is
> >> only one problem report. It doesn't break anything.  
> > 
> > Oh yes it breaks the fact that we do not need to care about the
> > value. If you hardcode one today you might be setting half the
> > default after the next apt-update. ... because it is a magic number
> > and not
> > 
> > set(get()*2)  
> 
> Unfortunately this option is not get-able. It's defined as member of 
> class in source code, which is initialized on class creation. There
> is no getter function to request this variable later.
> 
> 8<--
> apt-pkg/pkgcachegen.cc:
> 
> map_filesize_t const MapStart = _config->FindI("APT::Cache-Start", 
> 24*1024*1024);
> ...
> return new DynamicMMap(*CacheF, Flags, MapStart, MapGrow, MapLimit);
> 8<--
> 
> Anyway, I noticed that build fails later on any 'apt' call in 
> 'buildchroot' and 'image' for this configuration.
> 
> I was wrong with kernel version, it's 3.16 - the standard Jessie
> kernel. So for now, I'd like to drop this patch and add supported
> hosts matrix to Isar. This workaround could be described here.

I guess one could make such a workaround depending on "apt-get
--version", to prevent it from causing undesired things in versions
it is not for.

Henning

> Alex
> 
> >   
> >> We can't require Isar users to always have up-to-dated host
> >> systems. If this patch makes all Isar users with old kernels
> >> happy, it's probably good for Isar in general :-)  
> > 
> > Sure, but there is a cost ... see previous point.
> >     
> >>>
> >>> If this patch needs to go in, it is missing the cleanup step.
> >>>      
> >>
> >> Sorry, what do you mean here by cleanup step?  
> > 
> > Sorry, misread the patch. I thought it placed a file in the rootfs.
> > 
> > Henning
> >   
> >> Alex
> >>  
> >>> Henning
> >>>      
> >>>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> >>>>> ---
> >>>>>     meta/recipes-core/isar-bootstrap/isar-bootstrap.bb | 4 +++-
> >>>>>     1 file changed, 3 insertions(+), 1 deletion(-)
> >>>>>
> >>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb
> >>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb index
> >>>>> a38dd88..4cdefaa 100644 ---
> >>>>> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb +++
> >>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb @@ -187,7
> >>>>> +187,9 @@ do_apt_update() {
> >>>>>         E="${@bb.utils.export_proxies(d)}"
> >>>>>         export DEBIAN_FRONTEND=noninteractive
> >>>>> -    sudo -E chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y
> >>>>> +    sudo -E chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y \
> >>>>> +                                      -o
> >>>>> APT::Cache-Start=50331648 +
> >>>>>         sudo -E chroot "${ROOTFSDIR}" /usr/bin/apt-get
> >>>>> dist-upgrade -y \ -o Debug::pkgProblemResolver=yes
> >>>>>     }
> >>>>>         
> >>>>     
> >>>      
> >>  
> >   
>