From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6552866560629800960 X-Received: by 2002:a1c:7410:: with SMTP id p16-v6mr387103wmc.20.1525771356957; Tue, 08 May 2018 02:22:36 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a1c:e719:: with SMTP id e25-v6ls159010wmh.8.gmail; Tue, 08 May 2018 02:22:36 -0700 (PDT) X-Google-Smtp-Source: AB8JxZpvtkjDVjZilQkDMjHiWFrsgRHfRNdHMGXnCAu+iGSFHhP2plDoboJVJ6MW0M5XuLVH9D/r X-Received: by 2002:a1c:30c3:: with SMTP id w186-v6mr246537wmw.2.1525771356391; Tue, 08 May 2018 02:22:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1525771356; cv=none; d=google.com; s=arc-20160816; b=E6uVpvZjS4bwqNbDeFPnNebOnTHez5osL5Q7HCbHAoTcqo+iRfC9wItwSW7sh55bet ALIqDdoEZ9/XJ+7xDae3iFW81p4GMoXSbP3PSocfMsd0L4dYP1FV5+0sb2fEQKiYGLjQ 167loeKSg/jsGBRWB+qRriwzKD7eS6nEYEWq5Tnk3cEEgG20XMrzjKW8EKQ7fA3dZVaT FUhJG9cyeZCLvzl9Fy2iz8H/wXmOw+xLXYHN5uGPPRbMtMaACBRwiC5HpxhqfjdQZ3Xl KefSb6906P2okvNgpx7wtEga9YWkzhuBPz4RgOztSq6nsa6ZVS0ptKRwxAdNemlY+fv4 XWyw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:arc-authentication-results; bh=RUI/4eKEQybBGMaQogBrywUi7OyEZ9LmvVpDGTR/pIQ=; b=AtDjEPeELHy5aNO5tPU5mmpkBKbb1qjRgH1AoBy0hPSj6Ng7YZO919mx2xx8nyuIu0 06Kj6lBWwNwGXVvkl6zQrG91Udf2qDgjOQKgESE5ds1McCRYFMGDmMX+TQqgTfYSOgbT WiQxhnOH+aI/H8JZwt3XBXAgpRhuD7AR29gHMJafm2dMRhTEnxEHSS5MIJoz/LlKo82G lY5Vhjbf2P+g7v4NPksd5sdLZDGnrpqQjWvCIdKwg4CsO8hpT1P931X/XER7nysKvGqo uCmBvx9IdM3rNXK6ZhgRL2fOUFlAy1OMHRUaTmaG57PKnUkavw88N44y0SHTj1/QV1m5 lbSQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=henning.schild@siemens.com Return-Path: Received: from thoth.sbs.de (thoth.sbs.de. [192.35.17.2]) by gmr-mx.google.com with ESMTPS id v3si274018wmh.0.2018.05.08.02.22.36 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 08 May 2018 02:22:36 -0700 (PDT) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.2 as permitted sender) client-ip=192.35.17.2; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=henning.schild@siemens.com Received: from mail1.siemens.de (mail1.siemens.de [139.23.33.14]) by thoth.sbs.de (8.15.2/8.15.2) with ESMTPS id w489MZpT009223 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 8 May 2018 11:22:36 +0200 Received: from md1pvb1c.ad001.siemens.net (md1pvb1c.ad001.siemens.net [139.25.68.40]) by mail1.siemens.de (8.15.2/8.15.2) with ESMTP id w489MZAV025928; Tue, 8 May 2018 11:22:35 +0200 Date: Tue, 8 May 2018 11:22:33 +0200 From: Henning Schild To: Alexander Smirnov Cc: isar-users Subject: Re: [PATCH] isar-bootstrap: Increase cache room Message-ID: <20180508112233.73a78f0c@md1pvb1c.ad001.siemens.net> In-Reply-To: <3c4af798-d517-91c7-34b7-d733ff2eb421@ilbers.de> References: <20180507154836.25738-1-asmirnov@ilbers.de> <52bdb5c0-5f1f-5efc-5de3-9c5a9253ae3b@ilbers.de> <20180507183151.16ff9387@md1pvb1c.ad001.siemens.net> <95cd2859-b074-c8a2-4450-cae977ce9846@ilbers.de> <20180507191213.7b68192b@md1pvb1c.ad001.siemens.net> <6fbef55a-276d-43ca-3039-8556c4b1ad8c@ilbers.de> <20180508105625.1bc1ff88@md1pvb1c.ad001.siemens.net> <3c4af798-d517-91c7-34b7-d733ff2eb421@ilbers.de> X-Mailer: Claws Mail 3.15.0-dirty (GTK+ 2.24.31; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-TUID: OBfNqaIKVHvT Am Tue, 8 May 2018 12:07:55 +0300 schrieb Alexander Smirnov : > On 05/08/2018 11:56 AM, Henning Schild wrote: > > Am Tue, 8 May 2018 10:56:55 +0300 > > schrieb Alexander Smirnov : > > > >> On 05/07/2018 08:12 PM, Henning Schild wrote: > >>> Am Mon, 7 May 2018 19:48:16 +0300 > >>> schrieb Alexander Smirnov : > >>> > >>>> Hi, > >>>> > >>>> On 05/07/2018 07:31 PM, Henning Schild wrote: > >>>>> Am Mon, 7 May 2018 18:57:36 +0300 > >>>>> schrieb Alexander Smirnov : > >>>>> > >>>>>> On 05/07/2018 06:48 PM, Alexander Smirnov wrote: > >>>>>>> According to the man page for 'apt.conf', there are three > >>>>>>> variables: > >>>>>>> - Cache-Start: initial size of mmap cache room > >>>>>>> - Cache-Grow: gap for dynamic mmap cache growth > >>>>>>> - Cache-Limit: maximal cache size after growth > >>>>>>> > >>>>>>> If dynamic resize of mmap region is not avaialbe, the apt uses > >>>>>>> pre-allocated Cache-Start room [1] for mmap file. > >>>>>>> > >>>>>>> Building Isar on one of the Debian host with kernel 3.4, I got > >>>>>>> the following problem for 'qemuarm64-stretch' configuration: > >>>>>>> > >>>>>>> 8<-- > >>>>>>> Hit:1 http://security.debian.org stretch/updates InRelease > >>>>>>> Ign:2 http://ftp.de.debian.org/debian stretch InRelease > >>>>>>> Hit:3 http://ftp.de.debian.org/debian stretch-updates > >>>>>>> InRelease Hit:4 http://ftp.de.debian.org/debian stretch > >>>>>>> Release E: Dynamic MMap ran out of room. Please increase the > >>>>>>> size of APT::Cache-Start. Current value: 25165824. (man 5 > >>>>>>> apt.conf) qemu: uncaught target signal 11 (Segmentation > >>>>>>> fault) - core dumped Segmentation fault 8<-- > >>>>>>> > >>>>>>> I have no information, why exactly the room could not be > >>>>>>> re-sized on this system, but it would be better to increase > >>>>>>> the initial room size for apt. This patch increases the > >>>>>>> default apt cache twice. > >>>>>> > >>>>>> 1: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=657085 > >>>>>> > >>>>>> Missed :-( > >>>>> > >>>>> What is your conclusion from that link and what does it mean for > >>>>> the patch? > >>>>> > >>>> > >>>> This just the information I found about dynamic cache increasing. > >>>> Sometimes it could not work. Another fact is that increasing of > >>>> Cache-Limit doesn't help, so seems that mremap really doesn't > >>>> work. > >>>>> I do not like adding magic numbers and changing the default > >>>>> config, especially if the whole reason is to work around a > >>>>> problem only found on outdated build hosts (3.4 really?) > >>>> > >>>> I'm not 100% sure that kernel 3.4 is the source of the problem. > >>>> Tried to investigate, but for now without results. > >>>> > >>>> But if you try to google line: > >>>> > >>>> E: Dynamic MMap ran out of room. Please increase the size of > >>>> APT::Cache-Start > >>>> > >>>> There will be tons of search results, so such problem occurred > >>>> not only on my host. > >>>> > >>>> > >>>> The man page says: > >>>> > >>>> ...on systems with a lot of configured sources it should be > >>>> increased... > >>>> > >>>> so for me it makes sense to increase this room even if there is > >>>> only one problem report. It doesn't break anything. > >>> > >>> Oh yes it breaks the fact that we do not need to care about the > >>> value. If you hardcode one today you might be setting half the > >>> default after the next apt-update. ... because it is a magic > >>> number and not > >>> > >>> set(get()*2) > >> > >> Unfortunately this option is not get-able. It's defined as member > >> of class in source code, which is initialized on class creation. > >> There is no getter function to request this variable later. > >> > >> 8<-- > >> apt-pkg/pkgcachegen.cc: > >> > >> map_filesize_t const MapStart = _config->FindI("APT::Cache-Start", > >> 24*1024*1024); > >> ... > >> return new DynamicMMap(*CacheF, Flags, MapStart, MapGrow, > >> MapLimit); 8<-- > >> > >> Anyway, I noticed that build fails later on any 'apt' call in > >> 'buildchroot' and 'image' for this configuration. > >> > >> I was wrong with kernel version, it's 3.16 - the standard Jessie > >> kernel. So for now, I'd like to drop this patch and add supported > >> hosts matrix to Isar. This workaround could be described here. > > > > I guess one could make such a workaround depending on "apt-get > > --version", to prevent it from causing undesired things in versions > > it is not for. > > The problem occurred inside stretch chroot, while host kernel is from > jessie. So there should be check for pair: apt version - kernel > version: if kernel is old, apt is new - then fallback to array-based > cache. > > Does this look like a hack for you, or not? :-) Adding the magic-number argument is the real hack. The clearer we can pin that down to a specific setup, the better. So check the architecture and temperature as well, if it helps ;). Henning > Alex > > > > > Henning > > > >> Alex > >> > >>> > >>>> We can't require Isar users to always have up-to-dated host > >>>> systems. If this patch makes all Isar users with old kernels > >>>> happy, it's probably good for Isar in general :-) > >>> > >>> Sure, but there is a cost ... see previous point. > >>> > >>>>> > >>>>> If this patch needs to go in, it is missing the cleanup step. > >>>>> > >>>> > >>>> Sorry, what do you mean here by cleanup step? > >>> > >>> Sorry, misread the patch. I thought it placed a file in the > >>> rootfs. > >>> > >>> Henning > >>> > >>>> Alex > >>>> > >>>>> Henning > >>>>> > >>>>>>> Signed-off-by: Alexander Smirnov > >>>>>>> --- > >>>>>>> meta/recipes-core/isar-bootstrap/isar-bootstrap.bb | 4 > >>>>>>> +++- 1 file changed, 3 insertions(+), 1 deletion(-) > >>>>>>> > >>>>>>> diff --git > >>>>>>> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb > >>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb index > >>>>>>> a38dd88..4cdefaa 100644 --- > >>>>>>> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb +++ > >>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb @@ > >>>>>>> -187,7 +187,9 @@ do_apt_update() > >>>>>>> { E="${@bb.utils.export_proxies(d)}" export > >>>>>>> DEBIAN_FRONTEND=noninteractive > >>>>>>> - sudo -E chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y > >>>>>>> + sudo -E chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y > >>>>>>> \ > >>>>>>> + -o > >>>>>>> APT::Cache-Start=50331648 + > >>>>>>> sudo -E chroot "${ROOTFSDIR}" /usr/bin/apt-get > >>>>>>> dist-upgrade -y \ -o Debug::pkgProblemResolver=yes > >>>>>>> } > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > > >