From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6552866560629800960 X-Received: by 2002:a1c:eb07:: with SMTP id j7-v6mr387136wmh.19.1525770485250; Tue, 08 May 2018 02:08:05 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a1c:f707:: with SMTP id v7-v6ls142991wmh.1.canary-gmail; Tue, 08 May 2018 02:08:04 -0700 (PDT) X-Google-Smtp-Source: AB8JxZoRS368n4VTnvCXKyf5Z72B2ZchpDU3QTbSqYduSFBbAhyz3Xl09Rq4y/yjeTcb20sTBcRG X-Received: by 2002:a1c:820a:: with SMTP id e10-v6mr357743wmd.1.1525770484753; Tue, 08 May 2018 02:08:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1525770484; cv=none; d=google.com; s=arc-20160816; b=zqu1siLWm8yN5+FrwofhwIgClQ0BzPD8btrldD89C69FngkBdgPWaNdMbG5ZqRdcj2 E1KJbQxBbpHKkKKJ+NDs+yYQRC7P3u+yLbOrAhPWxz+ApGdCPxILA08s/wqOxtYfJDJu 6k3XyjldDoddnXQ5fInOI/oMiXIDYFHe3DVHr0RyUs4TjYq2valUVofwCKY8CmXvYlZi c28q+5O52SPWkGYyd088N00t8ts7Gu/SgClLgs9AY8OinYigW4CovBB2DYu5VOJfJps7 XeppOyeDmAecdYmPE67tB7aBGZh6ErdQeobn8bHfeY0gVYIUsFHp6e+bnKkBcnfF4ARc HLZg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject :arc-authentication-results; bh=Yg6+O+U5cJwwCVo7Ljawy3LWtPTXCZ99zUZO0s095N4=; b=KbyhvbEyNR7v9eoeo0LFbUvtMhDCH+Wzhz6lH5jaR4OG36/IV7e9opUPn5OwUm1ALT vJ49JEAIypWx0aajAQIc72diOGDha8SpErPGrrK8HIpOEHMZIWOvswdnMPNTTMDlknia XfsM8S4Hk3Dhc+nK9TsXLz63Ef/XAgSZFV+rHQCGeuYPllGY1W3s5LE0QlUH+mcoi6/t zCpa8bOLvmFZ7NJlys1TOxbYcH3yRZ10H6QLBFSk8yq7UKLSaxc4c9Q/3D9vpDLw7fQn Nv3wRd3Y6PqiYlIgP/drZv6LwgbpmF9ykOe2LtJblGSz2f5oIXpNIejEJS3mCsZfmPFT p8pA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id v12-v6si240501wmc.2.2018.05.08.02.08.04 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 08 May 2018 02:08:04 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Received: from [10.0.2.15] ([188.227.110.165]) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id w48981DD024012 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Tue, 8 May 2018 11:08:02 +0200 Subject: Re: [PATCH] isar-bootstrap: Increase cache room To: Henning Schild Cc: isar-users References: <20180507154836.25738-1-asmirnov@ilbers.de> <52bdb5c0-5f1f-5efc-5de3-9c5a9253ae3b@ilbers.de> <20180507183151.16ff9387@md1pvb1c.ad001.siemens.net> <95cd2859-b074-c8a2-4450-cae977ce9846@ilbers.de> <20180507191213.7b68192b@md1pvb1c.ad001.siemens.net> <6fbef55a-276d-43ca-3039-8556c4b1ad8c@ilbers.de> <20180508105625.1bc1ff88@md1pvb1c.ad001.siemens.net> From: Alexander Smirnov Message-ID: <3c4af798-d517-91c7-34b7-d733ff2eb421@ilbers.de> Date: Tue, 8 May 2018 12:07:55 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <20180508105625.1bc1ff88@md1pvb1c.ad001.siemens.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: gzroDyKixlBf On 05/08/2018 11:56 AM, Henning Schild wrote: > Am Tue, 8 May 2018 10:56:55 +0300 > schrieb Alexander Smirnov : > >> On 05/07/2018 08:12 PM, Henning Schild wrote: >>> Am Mon, 7 May 2018 19:48:16 +0300 >>> schrieb Alexander Smirnov : >>> >>>> Hi, >>>> >>>> On 05/07/2018 07:31 PM, Henning Schild wrote: >>>>> Am Mon, 7 May 2018 18:57:36 +0300 >>>>> schrieb Alexander Smirnov : >>>>> >>>>>> On 05/07/2018 06:48 PM, Alexander Smirnov wrote: >>>>>>> According to the man page for 'apt.conf', there are three >>>>>>> variables: >>>>>>> - Cache-Start: initial size of mmap cache room >>>>>>> - Cache-Grow: gap for dynamic mmap cache growth >>>>>>> - Cache-Limit: maximal cache size after growth >>>>>>> >>>>>>> If dynamic resize of mmap region is not avaialbe, the apt uses >>>>>>> pre-allocated Cache-Start room [1] for mmap file. >>>>>>> >>>>>>> Building Isar on one of the Debian host with kernel 3.4, I got >>>>>>> the following problem for 'qemuarm64-stretch' configuration: >>>>>>> >>>>>>> 8<-- >>>>>>> Hit:1 http://security.debian.org stretch/updates InRelease >>>>>>> Ign:2 http://ftp.de.debian.org/debian stretch InRelease >>>>>>> Hit:3 http://ftp.de.debian.org/debian stretch-updates InRelease >>>>>>> Hit:4 http://ftp.de.debian.org/debian stretch Release >>>>>>> E: Dynamic MMap ran out of room. Please increase the size of >>>>>>> APT::Cache-Start. Current value: 25165824. (man 5 apt.conf) >>>>>>> qemu: uncaught target signal 11 (Segmentation fault) - core >>>>>>> dumped Segmentation fault 8<-- >>>>>>> >>>>>>> I have no information, why exactly the room could not be >>>>>>> re-sized on this system, but it would be better to increase the >>>>>>> initial room size for apt. This patch increases the default apt >>>>>>> cache twice. >>>>>> >>>>>> 1: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=657085 >>>>>> >>>>>> Missed :-( >>>>> >>>>> What is your conclusion from that link and what does it mean for >>>>> the patch? >>>>> >>>> >>>> This just the information I found about dynamic cache increasing. >>>> Sometimes it could not work. Another fact is that increasing of >>>> Cache-Limit doesn't help, so seems that mremap really doesn't work. >>>> >>>>> I do not like adding magic numbers and changing the default >>>>> config, especially if the whole reason is to work around a >>>>> problem only found on outdated build hosts (3.4 really?) >>>> >>>> I'm not 100% sure that kernel 3.4 is the source of the problem. >>>> Tried to investigate, but for now without results. >>>> >>>> But if you try to google line: >>>> >>>> E: Dynamic MMap ran out of room. Please increase the size of >>>> APT::Cache-Start >>>> >>>> There will be tons of search results, so such problem occurred not >>>> only on my host. >>>> >>>> >>>> The man page says: >>>> >>>> ...on systems with a lot of configured sources it should be >>>> increased... >>>> >>>> so for me it makes sense to increase this room even if there is >>>> only one problem report. It doesn't break anything. >>> >>> Oh yes it breaks the fact that we do not need to care about the >>> value. If you hardcode one today you might be setting half the >>> default after the next apt-update. ... because it is a magic number >>> and not >>> >>> set(get()*2) >> >> Unfortunately this option is not get-able. It's defined as member of >> class in source code, which is initialized on class creation. There >> is no getter function to request this variable later. >> >> 8<-- >> apt-pkg/pkgcachegen.cc: >> >> map_filesize_t const MapStart = _config->FindI("APT::Cache-Start", >> 24*1024*1024); >> ... >> return new DynamicMMap(*CacheF, Flags, MapStart, MapGrow, MapLimit); >> 8<-- >> >> Anyway, I noticed that build fails later on any 'apt' call in >> 'buildchroot' and 'image' for this configuration. >> >> I was wrong with kernel version, it's 3.16 - the standard Jessie >> kernel. So for now, I'd like to drop this patch and add supported >> hosts matrix to Isar. This workaround could be described here. > > I guess one could make such a workaround depending on "apt-get > --version", to prevent it from causing undesired things in versions > it is not for. The problem occurred inside stretch chroot, while host kernel is from jessie. So there should be check for pair: apt version - kernel version: if kernel is old, apt is new - then fallback to array-based cache. Does this look like a hack for you, or not? :-) Alex > > Henning > >> Alex >> >>> >>>> We can't require Isar users to always have up-to-dated host >>>> systems. If this patch makes all Isar users with old kernels >>>> happy, it's probably good for Isar in general :-) >>> >>> Sure, but there is a cost ... see previous point. >>> >>>>> >>>>> If this patch needs to go in, it is missing the cleanup step. >>>>> >>>> >>>> Sorry, what do you mean here by cleanup step? >>> >>> Sorry, misread the patch. I thought it placed a file in the rootfs. >>> >>> Henning >>> >>>> Alex >>>> >>>>> Henning >>>>> >>>>>>> Signed-off-by: Alexander Smirnov >>>>>>> --- >>>>>>> meta/recipes-core/isar-bootstrap/isar-bootstrap.bb | 4 +++- >>>>>>> 1 file changed, 3 insertions(+), 1 deletion(-) >>>>>>> >>>>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb >>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb index >>>>>>> a38dd88..4cdefaa 100644 --- >>>>>>> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb +++ >>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb @@ -187,7 >>>>>>> +187,9 @@ do_apt_update() { >>>>>>> E="${@bb.utils.export_proxies(d)}" >>>>>>> export DEBIAN_FRONTEND=noninteractive >>>>>>> - sudo -E chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y >>>>>>> + sudo -E chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y \ >>>>>>> + -o >>>>>>> APT::Cache-Start=50331648 + >>>>>>> sudo -E chroot "${ROOTFSDIR}" /usr/bin/apt-get >>>>>>> dist-upgrade -y \ -o Debug::pkgProblemResolver=yes >>>>>>> } >>>>>>> >>>>>> >>>>> >>>> >>> >> > -- With best regards, Alexander Smirnov ilbers GmbH Baierbrunner Str. 28c D-81379 Munich +49 (89) 122 67 24-0 http://ilbers.de/ Commercial register Munich, HRB 214197 General manager: Baurzhan Ismagulov