From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6552866560629800960 X-Received: by 2002:a19:1441:: with SMTP id k62-v6mr447745lfi.11.1525775444676; Tue, 08 May 2018 03:30:44 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:9b07:: with SMTP id u7-v6ls102838lji.16.gmail; Tue, 08 May 2018 03:30:44 -0700 (PDT) X-Google-Smtp-Source: AB8JxZpc+atvWj6Uoo0drykywzn0Nv1GEvEmyEN1TDJotHftMAICI9TZ8UTWPVQqd65PessJ9ogH X-Received: by 2002:a2e:8086:: with SMTP id i6-v6mr1955030ljg.14.1525775444208; Tue, 08 May 2018 03:30:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1525775444; cv=none; d=google.com; s=arc-20160816; b=PSBgXY3xzCZZbNlarPm58g3mHLEygyqGhqVKRnf6H7IpQdFyZiNymlmC6NTzj/begL AM3ZltSMhcUoRnCHMoUMMNtU57uXm4vAaPn8pSQPFYlErAaCbY8BHQTCSQOpFIElvzdx 6AZtdWTegKF1F2jSCMvWg6ZndnBWH+VEC7RsjVdWfm19WSa9ifxScRaRaGh9PpRgG/cN 3Bd0SLMYtMgm+e3M5nhMAtdGDHqEe5c7YQkQwGmlCapcQY6ySOYTHimy56GDvSva4u/I /mLrwhZD3QgtrEZpvN1sPCw9GflEo/7ddse+6h3+JMmxyIIBSNtDf93Kfeqk6eqPa+oq ldQg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:arc-authentication-results; bh=XKH6/qzpFft+XTkt3KuB1yCeJGZX8+M4FNS8+GPDDGw=; b=uukYDX2w9ZG46cZNBlJkfLX0KsfbZ0rQMsaunyhuYRpDK96txfgNPmGN6s31JpQwbO oOzKs/y6G5py0pmJEbSSq7LFeRHhwOK5VqF6TtnVPoGlxOQMW0jSMMiQ1jVYKa0nVfK5 4VuiabGprjU009alaOSnM+BJNneFsYkh7Wa1UISLjPGBIx5HsHzk1CKA1VzOhyICVJVP ASyZl/EF8T+eC6PBMtUwb6gw+WrHPHpm+PgYrR/m+ALtB1zfHW8mOSjlX/LxWGDj+jrA zXl83tkQBNmh89OSY+TY3MHb2HVD8aovkqSZZvgKqYyImOfsMQDAqnU7yOnF41wpH6h+ OfMw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=henning.schild@siemens.com Return-Path: Received: from goliath.siemens.de (goliath.siemens.de. [192.35.17.28]) by gmr-mx.google.com with ESMTPS id j8-v6si171434ljh.3.2018.05.08.03.30.44 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 08 May 2018 03:30:44 -0700 (PDT) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.28 as permitted sender) client-ip=192.35.17.28; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=henning.schild@siemens.com Received: from mail1.siemens.de (mail1.siemens.de [139.23.33.14]) by goliath.siemens.de (8.15.2/8.15.2) with ESMTPS id w48AUh6Q010981 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 8 May 2018 12:30:43 +0200 Received: from md1pvb1c.ad001.siemens.net (md1pvb1c.ad001.siemens.net [139.25.68.40]) by mail1.siemens.de (8.15.2/8.15.2) with ESMTP id w48AUhYF009329; Tue, 8 May 2018 12:30:43 +0200 Date: Tue, 8 May 2018 12:30:42 +0200 From: Henning Schild To: Alexander Smirnov Cc: isar-users Subject: Re: [PATCH] isar-bootstrap: Increase cache room Message-ID: <20180508123042.05d233b3@md1pvb1c.ad001.siemens.net> In-Reply-To: <8e3f6a0b-77bd-b435-0424-910ee9f9d8a7@ilbers.de> References: <20180507154836.25738-1-asmirnov@ilbers.de> <52bdb5c0-5f1f-5efc-5de3-9c5a9253ae3b@ilbers.de> <20180507183151.16ff9387@md1pvb1c.ad001.siemens.net> <95cd2859-b074-c8a2-4450-cae977ce9846@ilbers.de> <20180507191213.7b68192b@md1pvb1c.ad001.siemens.net> <6fbef55a-276d-43ca-3039-8556c4b1ad8c@ilbers.de> <20180508105625.1bc1ff88@md1pvb1c.ad001.siemens.net> <3c4af798-d517-91c7-34b7-d733ff2eb421@ilbers.de> <20180508112233.73a78f0c@md1pvb1c.ad001.siemens.net> <8e3f6a0b-77bd-b435-0424-910ee9f9d8a7@ilbers.de> X-Mailer: Claws Mail 3.15.0-dirty (GTK+ 2.24.31; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-TUID: 3c0E6gqCk2e6 Am Tue, 8 May 2018 12:27:01 +0300 schrieb Alexander Smirnov : > On 05/08/2018 12:22 PM, Henning Schild wrote: > > Am Tue, 8 May 2018 12:07:55 +0300 > > schrieb Alexander Smirnov : > > > >> On 05/08/2018 11:56 AM, Henning Schild wrote: > >>> Am Tue, 8 May 2018 10:56:55 +0300 > >>> schrieb Alexander Smirnov : > >>> > >>>> On 05/07/2018 08:12 PM, Henning Schild wrote: > >>>>> Am Mon, 7 May 2018 19:48:16 +0300 > >>>>> schrieb Alexander Smirnov : > >>>>> > >>>>>> Hi, > >>>>>> > >>>>>> On 05/07/2018 07:31 PM, Henning Schild wrote: > >>>>>>> Am Mon, 7 May 2018 18:57:36 +0300 > >>>>>>> schrieb Alexander Smirnov : > >>>>>>> > >>>>>>>> On 05/07/2018 06:48 PM, Alexander Smirnov wrote: > >>>>>>>>> According to the man page for 'apt.conf', there are three > >>>>>>>>> variables: > >>>>>>>>> - Cache-Start: initial size of mmap cache room > >>>>>>>>> - Cache-Grow: gap for dynamic mmap cache growth > >>>>>>>>> - Cache-Limit: maximal cache size after growth > >>>>>>>>> > >>>>>>>>> If dynamic resize of mmap region is not avaialbe, the apt > >>>>>>>>> uses pre-allocated Cache-Start room [1] for mmap file. > >>>>>>>>> > >>>>>>>>> Building Isar on one of the Debian host with kernel 3.4, I > >>>>>>>>> got the following problem for 'qemuarm64-stretch' > >>>>>>>>> configuration: > >>>>>>>>> > >>>>>>>>> 8<-- > >>>>>>>>> Hit:1 http://security.debian.org stretch/updates InRelease > >>>>>>>>> Ign:2 http://ftp.de.debian.org/debian stretch InRelease > >>>>>>>>> Hit:3 http://ftp.de.debian.org/debian stretch-updates > >>>>>>>>> InRelease Hit:4 http://ftp.de.debian.org/debian stretch > >>>>>>>>> Release E: Dynamic MMap ran out of room. Please increase the > >>>>>>>>> size of APT::Cache-Start. Current value: 25165824. (man 5 > >>>>>>>>> apt.conf) qemu: uncaught target signal 11 (Segmentation > >>>>>>>>> fault) - core dumped Segmentation fault 8<-- > >>>>>>>>> > >>>>>>>>> I have no information, why exactly the room could not be > >>>>>>>>> re-sized on this system, but it would be better to increase > >>>>>>>>> the initial room size for apt. This patch increases the > >>>>>>>>> default apt cache twice. > >>>>>>>> > >>>>>>>> 1: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=657085 > >>>>>>>> > >>>>>>>> Missed :-( > >>>>>>> > >>>>>>> What is your conclusion from that link and what does it mean > >>>>>>> for the patch? > >>>>>>> > >>>>>> > >>>>>> This just the information I found about dynamic cache > >>>>>> increasing. Sometimes it could not work. Another fact is that > >>>>>> increasing of Cache-Limit doesn't help, so seems that mremap > >>>>>> really doesn't work. > >>>>>>> I do not like adding magic numbers and changing the default > >>>>>>> config, especially if the whole reason is to work around a > >>>>>>> problem only found on outdated build hosts (3.4 really?) > >>>>>> > >>>>>> I'm not 100% sure that kernel 3.4 is the source of the problem. > >>>>>> Tried to investigate, but for now without results. > >>>>>> > >>>>>> But if you try to google line: > >>>>>> > >>>>>> E: Dynamic MMap ran out of room. Please increase the > >>>>>> size of APT::Cache-Start > >>>>>> > >>>>>> There will be tons of search results, so such problem occurred > >>>>>> not only on my host. > >>>>>> > >>>>>> > >>>>>> The man page says: > >>>>>> > >>>>>> ...on systems with a lot of configured sources it should be > >>>>>> increased... > >>>>>> > >>>>>> so for me it makes sense to increase this room even if there is > >>>>>> only one problem report. It doesn't break anything. > >>>>> > >>>>> Oh yes it breaks the fact that we do not need to care about the > >>>>> value. If you hardcode one today you might be setting half the > >>>>> default after the next apt-update. ... because it is a magic > >>>>> number and not > >>>>> > >>>>> set(get()*2) > >>>> > >>>> Unfortunately this option is not get-able. It's defined as member > >>>> of class in source code, which is initialized on class creation. > >>>> There is no getter function to request this variable later. > >>>> > >>>> 8<-- > >>>> apt-pkg/pkgcachegen.cc: > >>>> > >>>> map_filesize_t const MapStart = > >>>> _config->FindI("APT::Cache-Start", 24*1024*1024); > >>>> ... > >>>> return new DynamicMMap(*CacheF, Flags, MapStart, MapGrow, > >>>> MapLimit); 8<-- > >>>> > >>>> Anyway, I noticed that build fails later on any 'apt' call in > >>>> 'buildchroot' and 'image' for this configuration. > >>>> > >>>> I was wrong with kernel version, it's 3.16 - the standard Jessie > >>>> kernel. So for now, I'd like to drop this patch and add supported > >>>> hosts matrix to Isar. This workaround could be described here. > >>> > >>> I guess one could make such a workaround depending on "apt-get > >>> --version", to prevent it from causing undesired things in > >>> versions it is not for. > >> > >> The problem occurred inside stretch chroot, while host kernel is > >> from jessie. So there should be check for pair: apt version - > >> kernel version: if kernel is old, apt is new - then fallback to > >> array-based cache. > >> > >> Does this look like a hack for you, or not? :-) > > > > Adding the magic-number argument is the real hack. The clearer we > > can pin that down to a specific setup, the better. So check the > > architecture and temperature as well, if it helps ;). > > No magic numbers, the parameter: -o APT::Cache-Fallback="1" works as > well. > > So, thing like: > > if target=stretch and uname=old; then > apt-fallback=1 > fi > > would be OK from your POV? Yes. Henning > Alex > > > > > Henning > > > >> Alex > >> > >>> > >>> Henning > >>> > >>>> Alex > >>>> > >>>>> > >>>>>> We can't require Isar users to always have up-to-dated host > >>>>>> systems. If this patch makes all Isar users with old kernels > >>>>>> happy, it's probably good for Isar in general :-) > >>>>> > >>>>> Sure, but there is a cost ... see previous point. > >>>>> > >>>>>>> > >>>>>>> If this patch needs to go in, it is missing the cleanup step. > >>>>>>> > >>>>>> > >>>>>> Sorry, what do you mean here by cleanup step? > >>>>> > >>>>> Sorry, misread the patch. I thought it placed a file in the > >>>>> rootfs. > >>>>> > >>>>> Henning > >>>>> > >>>>>> Alex > >>>>>> > >>>>>>> Henning > >>>>>>> > >>>>>>>>> Signed-off-by: Alexander Smirnov > >>>>>>>>> --- > >>>>>>>>> meta/recipes-core/isar-bootstrap/isar-bootstrap.bb | 4 > >>>>>>>>> +++- 1 file changed, 3 insertions(+), 1 deletion(-) > >>>>>>>>> > >>>>>>>>> diff --git > >>>>>>>>> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb > >>>>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb index > >>>>>>>>> a38dd88..4cdefaa 100644 --- > >>>>>>>>> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb +++ > >>>>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb @@ > >>>>>>>>> -187,7 +187,9 @@ do_apt_update() > >>>>>>>>> { E="${@bb.utils.export_proxies(d)}" export > >>>>>>>>> DEBIAN_FRONTEND=noninteractive > >>>>>>>>> - sudo -E chroot "${ROOTFSDIR}" /usr/bin/apt-get update > >>>>>>>>> -y > >>>>>>>>> + sudo -E chroot "${ROOTFSDIR}" /usr/bin/apt-get update > >>>>>>>>> -y \ > >>>>>>>>> + -o > >>>>>>>>> APT::Cache-Start=50331648 + > >>>>>>>>> sudo -E chroot "${ROOTFSDIR}" /usr/bin/apt-get > >>>>>>>>> dist-upgrade -y \ -o Debug::pkgProblemResolver=yes > >>>>>>>>> } > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > > >