Re: [PATCH] isar-bootstrap: Increase cache room

[Alexander Smirnov <asmirnov@ilbers.de>]

On 05/08/2018 11:56 AM, Henning Schild wrote:
> Am Tue, 8 May 2018 10:56:55 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> On 05/07/2018 08:12 PM, Henning Schild wrote:
>>> Am Mon, 7 May 2018 19:48:16 +0300
>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>    
>>>> Hi,
>>>>
>>>> On 05/07/2018 07:31 PM, Henning Schild wrote:
>>>>> Am Mon, 7 May 2018 18:57:36 +0300
>>>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>>>       
>>>>>> On 05/07/2018 06:48 PM, Alexander Smirnov wrote:
>>>>>>> According to the man page for 'apt.conf', there are three
>>>>>>> variables:
>>>>>>>      - Cache-Start: initial size of mmap cache room
>>>>>>>      - Cache-Grow: gap for dynamic mmap cache growth
>>>>>>>      - Cache-Limit: maximal cache size after growth
>>>>>>>
>>>>>>> If dynamic resize of mmap region is not avaialbe, the apt uses
>>>>>>> pre-allocated Cache-Start room [1] for mmap file.
>>>>>>>
>>>>>>> Building Isar on one of the Debian host with kernel 3.4, I got
>>>>>>> the following problem for 'qemuarm64-stretch' configuration:
>>>>>>>
>>>>>>> 8<--
>>>>>>> Hit:1 http://security.debian.org stretch/updates InRelease
>>>>>>> Ign:2 http://ftp.de.debian.org/debian stretch InRelease
>>>>>>> Hit:3 http://ftp.de.debian.org/debian stretch-updates InRelease
>>>>>>> Hit:4 http://ftp.de.debian.org/debian stretch Release
>>>>>>> E: Dynamic MMap ran out of room. Please increase the size of
>>>>>>> APT::Cache-Start. Current value: 25165824. (man 5 apt.conf)
>>>>>>> qemu: uncaught target signal 11 (Segmentation fault) - core
>>>>>>> dumped Segmentation fault 8<--
>>>>>>>
>>>>>>> I have no information, why exactly the room could not be
>>>>>>> re-sized on this system, but it would be better to increase the
>>>>>>> initial room size for apt. This patch increases the default apt
>>>>>>> cache twice.
>>>>>>
>>>>>> 1: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=657085
>>>>>>
>>>>>> Missed :-(
>>>>>
>>>>> What is your conclusion from that link and what does it mean for
>>>>> the patch?
>>>>>       
>>>>
>>>> This just the information I found about dynamic cache increasing.
>>>> Sometimes it could not work. Another fact is that increasing of
>>>> Cache-Limit doesn't help, so seems that mremap really doesn't work.
>>>>   
>>>>> I do not like adding magic numbers and changing the default
>>>>> config, especially if the whole reason is to work around a
>>>>> problem only found on outdated build hosts (3.4 really?)
>>>>
>>>> I'm not 100% sure that kernel 3.4 is the source of the problem.
>>>> Tried to investigate, but for now without results.
>>>>
>>>> But if you try to google line:
>>>>
>>>>      E: Dynamic MMap ran out of room. Please increase the size of
>>>> APT::Cache-Start
>>>>
>>>> There will be tons of search results, so such problem occurred not
>>>> only on my host.
>>>>
>>>>
>>>> The man page says:
>>>>
>>>> ...on systems with a lot of configured sources it should be
>>>> increased...
>>>>
>>>> so for me it makes sense to increase this room even if there is
>>>> only one problem report. It doesn't break anything.
>>>
>>> Oh yes it breaks the fact that we do not need to care about the
>>> value. If you hardcode one today you might be setting half the
>>> default after the next apt-update. ... because it is a magic number
>>> and not
>>>
>>> set(get()*2)
>>
>> Unfortunately this option is not get-able. It's defined as member of
>> class in source code, which is initialized on class creation. There
>> is no getter function to request this variable later.
>>
>> 8<--
>> apt-pkg/pkgcachegen.cc:
>>
>> map_filesize_t const MapStart = _config->FindI("APT::Cache-Start",
>> 24*1024*1024);
>> ...
>> return new DynamicMMap(*CacheF, Flags, MapStart, MapGrow, MapLimit);
>> 8<--
>>
>> Anyway, I noticed that build fails later on any 'apt' call in
>> 'buildchroot' and 'image' for this configuration.
>>
>> I was wrong with kernel version, it's 3.16 - the standard Jessie
>> kernel. So for now, I'd like to drop this patch and add supported
>> hosts matrix to Isar. This workaround could be described here.
> 
> I guess one could make such a workaround depending on "apt-get
> --version", to prevent it from causing undesired things in versions
> it is not for.

The problem occurred inside stretch chroot, while host kernel is from 
jessie. So there should be check for pair: apt version - kernel version: 
  if kernel is old, apt is new - then fallback to array-based cache.

Does this look like a hack for you, or not? :-)

Alex

> 
> Henning
> 
>> Alex
>>
>>>    
>>>> We can't require Isar users to always have up-to-dated host
>>>> systems. If this patch makes all Isar users with old kernels
>>>> happy, it's probably good for Isar in general :-)
>>>
>>> Sure, but there is a cost ... see previous point.
>>>      
>>>>>
>>>>> If this patch needs to go in, it is missing the cleanup step.
>>>>>       
>>>>
>>>> Sorry, what do you mean here by cleanup step?
>>>
>>> Sorry, misread the patch. I thought it placed a file in the rootfs.
>>>
>>> Henning
>>>    
>>>> Alex
>>>>   
>>>>> Henning
>>>>>       
>>>>>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>>>>>>> ---
>>>>>>>      meta/recipes-core/isar-bootstrap/isar-bootstrap.bb | 4 +++-
>>>>>>>      1 file changed, 3 insertions(+), 1 deletion(-)
>>>>>>>
>>>>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb
>>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb index
>>>>>>> a38dd88..4cdefaa 100644 ---
>>>>>>> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb +++
>>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb @@ -187,7
>>>>>>> +187,9 @@ do_apt_update() {
>>>>>>>          E="${@bb.utils.export_proxies(d)}"
>>>>>>>          export DEBIAN_FRONTEND=noninteractive
>>>>>>> -    sudo -E chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y
>>>>>>> +    sudo -E chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y \
>>>>>>> +                                      -o
>>>>>>> APT::Cache-Start=50331648 +
>>>>>>>          sudo -E chroot "${ROOTFSDIR}" /usr/bin/apt-get
>>>>>>> dist-upgrade -y \ -o Debug::pkgProblemResolver=yes
>>>>>>>      }
>>>>>>>          
>>>>>>      
>>>>>       
>>>>   
>>>    
>>
> 

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov