From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6552866560629800960 X-Received: by 2002:adf:e18b:: with SMTP id k11-v6mr3581772wri.32.1525771631035; Tue, 08 May 2018 02:27:11 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a1c:e642:: with SMTP id d63-v6ls167649wmh.0.gmail; Tue, 08 May 2018 02:27:10 -0700 (PDT) X-Google-Smtp-Source: AB8JxZqUSfGy2sasKZsOYNl/fPiN5l5vcmEpW4vORLzU8Br/SOTbRhno/8HnCDJqQ27PWjq5ywDO X-Received: by 2002:a1c:a950:: with SMTP id s77-v6mr374673wme.27.1525771630516; Tue, 08 May 2018 02:27:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1525771630; cv=none; d=google.com; s=arc-20160816; b=zC18hav+gV+Eh8WnmUBiIFCtHdzP8OCFW5Pfyk6X1I2Y0o/WhrMHcj0ONUymkHdkR/ RDXBJaYubBZ+jBRINSVw4oFi6TRO5/2HC8Shstj4MdKutFWuA3zH85ptR0sr5Gdd2xra FcXrZAh1swian5phRNmFqGwGTsX79jm6oLkOCDhPP6NvO+smGPwLzWvh3Spcuv5wV5Sa 6sctLODzQFUx5iS2ILlmDmxxLkDtHYENhltvfDTA4FjFMNDcjp8I2dg3NLvjsziQPJkp ms8mU5Vn+Qr8M5dL7xnxhzmHEnfJKL7ZdJDiWvDF4lyafSPTbzxACNZVWqhbkUpcYcLS ajpA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject :arc-authentication-results; bh=BKpOmV50r1XHmbV5HTzSA3c8PEIF+UEu9Fxz+R4lncU=; b=efkRHuSMJGU2QiQ+C/A9mtSYd1w5gDNyV8ufFqA0xz8Ljq1yQemjEv93U/n+YYnJk9 I1LSYrmozmgIhrm/JMJ0qKjHxcSAHXFJxtlO6paH618jC8iDDXUhiHQlWM44Hgtvmxx8 kmFv0EyGKM+PrSJsOxEky37GQpzlASNVKWfUaJU4wMb5j+eKGshq+oNvuetjpVcr5pjk kO+gfkIeVKliXFbDNmTpDAHFseSiW92UD+Ld4kffn+Rqv3c6yub/77w8cP/Lq+06x5CV e203opdC3mPJ7LwhharBxz0ofAFefVy4IvucT8G3JfwSxZySxpldSc0waDAKnFh8y9ab wsIg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id n9-v6si905352wri.5.2018.05.08.02.27.10 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 08 May 2018 02:27:10 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Received: from [10.0.2.15] ([188.227.110.165]) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id w489R7lA024060 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Tue, 8 May 2018 11:27:09 +0200 Subject: Re: [PATCH] isar-bootstrap: Increase cache room To: Henning Schild Cc: isar-users References: <20180507154836.25738-1-asmirnov@ilbers.de> <52bdb5c0-5f1f-5efc-5de3-9c5a9253ae3b@ilbers.de> <20180507183151.16ff9387@md1pvb1c.ad001.siemens.net> <95cd2859-b074-c8a2-4450-cae977ce9846@ilbers.de> <20180507191213.7b68192b@md1pvb1c.ad001.siemens.net> <6fbef55a-276d-43ca-3039-8556c4b1ad8c@ilbers.de> <20180508105625.1bc1ff88@md1pvb1c.ad001.siemens.net> <3c4af798-d517-91c7-34b7-d733ff2eb421@ilbers.de> <20180508112233.73a78f0c@md1pvb1c.ad001.siemens.net> From: Alexander Smirnov Message-ID: <8e3f6a0b-77bd-b435-0424-910ee9f9d8a7@ilbers.de> Date: Tue, 8 May 2018 12:27:01 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <20180508112233.73a78f0c@md1pvb1c.ad001.siemens.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: BByZbZCdqhJY On 05/08/2018 12:22 PM, Henning Schild wrote: > Am Tue, 8 May 2018 12:07:55 +0300 > schrieb Alexander Smirnov : > >> On 05/08/2018 11:56 AM, Henning Schild wrote: >>> Am Tue, 8 May 2018 10:56:55 +0300 >>> schrieb Alexander Smirnov : >>> >>>> On 05/07/2018 08:12 PM, Henning Schild wrote: >>>>> Am Mon, 7 May 2018 19:48:16 +0300 >>>>> schrieb Alexander Smirnov : >>>>> >>>>>> Hi, >>>>>> >>>>>> On 05/07/2018 07:31 PM, Henning Schild wrote: >>>>>>> Am Mon, 7 May 2018 18:57:36 +0300 >>>>>>> schrieb Alexander Smirnov : >>>>>>> >>>>>>>> On 05/07/2018 06:48 PM, Alexander Smirnov wrote: >>>>>>>>> According to the man page for 'apt.conf', there are three >>>>>>>>> variables: >>>>>>>>> - Cache-Start: initial size of mmap cache room >>>>>>>>> - Cache-Grow: gap for dynamic mmap cache growth >>>>>>>>> - Cache-Limit: maximal cache size after growth >>>>>>>>> >>>>>>>>> If dynamic resize of mmap region is not avaialbe, the apt uses >>>>>>>>> pre-allocated Cache-Start room [1] for mmap file. >>>>>>>>> >>>>>>>>> Building Isar on one of the Debian host with kernel 3.4, I got >>>>>>>>> the following problem for 'qemuarm64-stretch' configuration: >>>>>>>>> >>>>>>>>> 8<-- >>>>>>>>> Hit:1 http://security.debian.org stretch/updates InRelease >>>>>>>>> Ign:2 http://ftp.de.debian.org/debian stretch InRelease >>>>>>>>> Hit:3 http://ftp.de.debian.org/debian stretch-updates >>>>>>>>> InRelease Hit:4 http://ftp.de.debian.org/debian stretch >>>>>>>>> Release E: Dynamic MMap ran out of room. Please increase the >>>>>>>>> size of APT::Cache-Start. Current value: 25165824. (man 5 >>>>>>>>> apt.conf) qemu: uncaught target signal 11 (Segmentation >>>>>>>>> fault) - core dumped Segmentation fault 8<-- >>>>>>>>> >>>>>>>>> I have no information, why exactly the room could not be >>>>>>>>> re-sized on this system, but it would be better to increase >>>>>>>>> the initial room size for apt. This patch increases the >>>>>>>>> default apt cache twice. >>>>>>>> >>>>>>>> 1: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=657085 >>>>>>>> >>>>>>>> Missed :-( >>>>>>> >>>>>>> What is your conclusion from that link and what does it mean for >>>>>>> the patch? >>>>>>> >>>>>> >>>>>> This just the information I found about dynamic cache increasing. >>>>>> Sometimes it could not work. Another fact is that increasing of >>>>>> Cache-Limit doesn't help, so seems that mremap really doesn't >>>>>> work. >>>>>>> I do not like adding magic numbers and changing the default >>>>>>> config, especially if the whole reason is to work around a >>>>>>> problem only found on outdated build hosts (3.4 really?) >>>>>> >>>>>> I'm not 100% sure that kernel 3.4 is the source of the problem. >>>>>> Tried to investigate, but for now without results. >>>>>> >>>>>> But if you try to google line: >>>>>> >>>>>> E: Dynamic MMap ran out of room. Please increase the size of >>>>>> APT::Cache-Start >>>>>> >>>>>> There will be tons of search results, so such problem occurred >>>>>> not only on my host. >>>>>> >>>>>> >>>>>> The man page says: >>>>>> >>>>>> ...on systems with a lot of configured sources it should be >>>>>> increased... >>>>>> >>>>>> so for me it makes sense to increase this room even if there is >>>>>> only one problem report. It doesn't break anything. >>>>> >>>>> Oh yes it breaks the fact that we do not need to care about the >>>>> value. If you hardcode one today you might be setting half the >>>>> default after the next apt-update. ... because it is a magic >>>>> number and not >>>>> >>>>> set(get()*2) >>>> >>>> Unfortunately this option is not get-able. It's defined as member >>>> of class in source code, which is initialized on class creation. >>>> There is no getter function to request this variable later. >>>> >>>> 8<-- >>>> apt-pkg/pkgcachegen.cc: >>>> >>>> map_filesize_t const MapStart = _config->FindI("APT::Cache-Start", >>>> 24*1024*1024); >>>> ... >>>> return new DynamicMMap(*CacheF, Flags, MapStart, MapGrow, >>>> MapLimit); 8<-- >>>> >>>> Anyway, I noticed that build fails later on any 'apt' call in >>>> 'buildchroot' and 'image' for this configuration. >>>> >>>> I was wrong with kernel version, it's 3.16 - the standard Jessie >>>> kernel. So for now, I'd like to drop this patch and add supported >>>> hosts matrix to Isar. This workaround could be described here. >>> >>> I guess one could make such a workaround depending on "apt-get >>> --version", to prevent it from causing undesired things in versions >>> it is not for. >> >> The problem occurred inside stretch chroot, while host kernel is from >> jessie. So there should be check for pair: apt version - kernel >> version: if kernel is old, apt is new - then fallback to array-based >> cache. >> >> Does this look like a hack for you, or not? :-) > > Adding the magic-number argument is the real hack. The clearer we can > pin that down to a specific setup, the better. So check the > architecture and temperature as well, if it helps ;). No magic numbers, the parameter: -o APT::Cache-Fallback="1" works as well. So, thing like: if target=stretch and uname=old; then apt-fallback=1 fi would be OK from your POV? Alex > > Henning > >> Alex >> >>> >>> Henning >>> >>>> Alex >>>> >>>>> >>>>>> We can't require Isar users to always have up-to-dated host >>>>>> systems. If this patch makes all Isar users with old kernels >>>>>> happy, it's probably good for Isar in general :-) >>>>> >>>>> Sure, but there is a cost ... see previous point. >>>>> >>>>>>> >>>>>>> If this patch needs to go in, it is missing the cleanup step. >>>>>>> >>>>>> >>>>>> Sorry, what do you mean here by cleanup step? >>>>> >>>>> Sorry, misread the patch. I thought it placed a file in the >>>>> rootfs. >>>>> >>>>> Henning >>>>> >>>>>> Alex >>>>>> >>>>>>> Henning >>>>>>> >>>>>>>>> Signed-off-by: Alexander Smirnov >>>>>>>>> --- >>>>>>>>> meta/recipes-core/isar-bootstrap/isar-bootstrap.bb | 4 >>>>>>>>> +++- 1 file changed, 3 insertions(+), 1 deletion(-) >>>>>>>>> >>>>>>>>> diff --git >>>>>>>>> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb >>>>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb index >>>>>>>>> a38dd88..4cdefaa 100644 --- >>>>>>>>> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb +++ >>>>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb @@ >>>>>>>>> -187,7 +187,9 @@ do_apt_update() >>>>>>>>> { E="${@bb.utils.export_proxies(d)}" export >>>>>>>>> DEBIAN_FRONTEND=noninteractive >>>>>>>>> - sudo -E chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y >>>>>>>>> + sudo -E chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y >>>>>>>>> \ >>>>>>>>> + -o >>>>>>>>> APT::Cache-Start=50331648 + >>>>>>>>> sudo -E chroot "${ROOTFSDIR}" /usr/bin/apt-get >>>>>>>>> dist-upgrade -y \ -o Debug::pkgProblemResolver=yes >>>>>>>>> } >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -- With best regards, Alexander Smirnov ilbers GmbH Baierbrunner Str. 28c D-81379 Munich +49 (89) 122 67 24-0 http://ilbers.de/ Commercial register Munich, HRB 214197 General manager: Baurzhan Ismagulov