Re: [PATCH] isar-bootstrap: Increase cache room

[Alexander Smirnov <asmirnov@ilbers.de>]

On 05/08/2018 12:22 PM, Henning Schild wrote:
> Am Tue, 8 May 2018 12:07:55 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> On 05/08/2018 11:56 AM, Henning Schild wrote:
>>> Am Tue, 8 May 2018 10:56:55 +0300
>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>    
>>>> On 05/07/2018 08:12 PM, Henning Schild wrote:
>>>>> Am Mon, 7 May 2018 19:48:16 +0300
>>>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>>>       
>>>>>> Hi,
>>>>>>
>>>>>> On 05/07/2018 07:31 PM, Henning Schild wrote:
>>>>>>> Am Mon, 7 May 2018 18:57:36 +0300
>>>>>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>>>>>          
>>>>>>>> On 05/07/2018 06:48 PM, Alexander Smirnov wrote:
>>>>>>>>> According to the man page for 'apt.conf', there are three
>>>>>>>>> variables:
>>>>>>>>>       - Cache-Start: initial size of mmap cache room
>>>>>>>>>       - Cache-Grow: gap for dynamic mmap cache growth
>>>>>>>>>       - Cache-Limit: maximal cache size after growth
>>>>>>>>>
>>>>>>>>> If dynamic resize of mmap region is not avaialbe, the apt uses
>>>>>>>>> pre-allocated Cache-Start room [1] for mmap file.
>>>>>>>>>
>>>>>>>>> Building Isar on one of the Debian host with kernel 3.4, I got
>>>>>>>>> the following problem for 'qemuarm64-stretch' configuration:
>>>>>>>>>
>>>>>>>>> 8<--
>>>>>>>>> Hit:1 http://security.debian.org stretch/updates InRelease
>>>>>>>>> Ign:2 http://ftp.de.debian.org/debian stretch InRelease
>>>>>>>>> Hit:3 http://ftp.de.debian.org/debian stretch-updates
>>>>>>>>> InRelease Hit:4 http://ftp.de.debian.org/debian stretch
>>>>>>>>> Release E: Dynamic MMap ran out of room. Please increase the
>>>>>>>>> size of APT::Cache-Start. Current value: 25165824. (man 5
>>>>>>>>> apt.conf) qemu: uncaught target signal 11 (Segmentation
>>>>>>>>> fault) - core dumped Segmentation fault 8<--
>>>>>>>>>
>>>>>>>>> I have no information, why exactly the room could not be
>>>>>>>>> re-sized on this system, but it would be better to increase
>>>>>>>>> the initial room size for apt. This patch increases the
>>>>>>>>> default apt cache twice.
>>>>>>>>
>>>>>>>> 1: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=657085
>>>>>>>>
>>>>>>>> Missed :-(
>>>>>>>
>>>>>>> What is your conclusion from that link and what does it mean for
>>>>>>> the patch?
>>>>>>>          
>>>>>>
>>>>>> This just the information I found about dynamic cache increasing.
>>>>>> Sometimes it could not work. Another fact is that increasing of
>>>>>> Cache-Limit doesn't help, so seems that mremap really doesn't
>>>>>> work.
>>>>>>> I do not like adding magic numbers and changing the default
>>>>>>> config, especially if the whole reason is to work around a
>>>>>>> problem only found on outdated build hosts (3.4 really?)
>>>>>>
>>>>>> I'm not 100% sure that kernel 3.4 is the source of the problem.
>>>>>> Tried to investigate, but for now without results.
>>>>>>
>>>>>> But if you try to google line:
>>>>>>
>>>>>>       E: Dynamic MMap ran out of room. Please increase the size of
>>>>>> APT::Cache-Start
>>>>>>
>>>>>> There will be tons of search results, so such problem occurred
>>>>>> not only on my host.
>>>>>>
>>>>>>
>>>>>> The man page says:
>>>>>>
>>>>>> ...on systems with a lot of configured sources it should be
>>>>>> increased...
>>>>>>
>>>>>> so for me it makes sense to increase this room even if there is
>>>>>> only one problem report. It doesn't break anything.
>>>>>
>>>>> Oh yes it breaks the fact that we do not need to care about the
>>>>> value. If you hardcode one today you might be setting half the
>>>>> default after the next apt-update. ... because it is a magic
>>>>> number and not
>>>>>
>>>>> set(get()*2)
>>>>
>>>> Unfortunately this option is not get-able. It's defined as member
>>>> of class in source code, which is initialized on class creation.
>>>> There is no getter function to request this variable later.
>>>>
>>>> 8<--
>>>> apt-pkg/pkgcachegen.cc:
>>>>
>>>> map_filesize_t const MapStart = _config->FindI("APT::Cache-Start",
>>>> 24*1024*1024);
>>>> ...
>>>> return new DynamicMMap(*CacheF, Flags, MapStart, MapGrow,
>>>> MapLimit); 8<--
>>>>
>>>> Anyway, I noticed that build fails later on any 'apt' call in
>>>> 'buildchroot' and 'image' for this configuration.
>>>>
>>>> I was wrong with kernel version, it's 3.16 - the standard Jessie
>>>> kernel. So for now, I'd like to drop this patch and add supported
>>>> hosts matrix to Isar. This workaround could be described here.
>>>
>>> I guess one could make such a workaround depending on "apt-get
>>> --version", to prevent it from causing undesired things in versions
>>> it is not for.
>>
>> The problem occurred inside stretch chroot, while host kernel is from
>> jessie. So there should be check for pair: apt version - kernel
>> version: if kernel is old, apt is new - then fallback to array-based
>> cache.
>>
>> Does this look like a hack for you, or not? :-)
> 
> Adding the magic-number argument is the real hack. The clearer we can
> pin that down to a specific setup, the better. So check the
> architecture and temperature as well, if it helps ;).

No magic numbers, the parameter: -o APT::Cache-Fallback="1" works as well.

So, thing like:

if target=stretch and uname=old; then
    apt-fallback=1
fi

would be OK from your POV?

Alex

> 
> Henning
> 
>> Alex
>>
>>>
>>> Henning
>>>    
>>>> Alex
>>>>   
>>>>>       
>>>>>> We can't require Isar users to always have up-to-dated host
>>>>>> systems. If this patch makes all Isar users with old kernels
>>>>>> happy, it's probably good for Isar in general :-)
>>>>>
>>>>> Sure, but there is a cost ... see previous point.
>>>>>         
>>>>>>>
>>>>>>> If this patch needs to go in, it is missing the cleanup step.
>>>>>>>          
>>>>>>
>>>>>> Sorry, what do you mean here by cleanup step?
>>>>>
>>>>> Sorry, misread the patch. I thought it placed a file in the
>>>>> rootfs.
>>>>>
>>>>> Henning
>>>>>       
>>>>>> Alex
>>>>>>      
>>>>>>> Henning
>>>>>>>          
>>>>>>>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>>>>>>>>> ---
>>>>>>>>>       meta/recipes-core/isar-bootstrap/isar-bootstrap.bb | 4
>>>>>>>>> +++- 1 file changed, 3 insertions(+), 1 deletion(-)
>>>>>>>>>
>>>>>>>>> diff --git
>>>>>>>>> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb
>>>>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb index
>>>>>>>>> a38dd88..4cdefaa 100644 ---
>>>>>>>>> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb +++
>>>>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb @@
>>>>>>>>> -187,7 +187,9 @@ do_apt_update()
>>>>>>>>> { E="${@bb.utils.export_proxies(d)}" export
>>>>>>>>> DEBIAN_FRONTEND=noninteractive
>>>>>>>>> -    sudo -E chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y
>>>>>>>>> +    sudo -E chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y
>>>>>>>>> \
>>>>>>>>> +                                      -o
>>>>>>>>> APT::Cache-Start=50331648 +
>>>>>>>>>           sudo -E chroot "${ROOTFSDIR}" /usr/bin/apt-get
>>>>>>>>> dist-upgrade -y \ -o Debug::pkgProblemResolver=yes
>>>>>>>>>       }
>>>>>>>>>             
>>>>>>>>         
>>>>>>>          
>>>>>>      
>>>>>       
>>>>   
>>>    
>>
> 

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov