From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6467463440282681344 X-Received: by 10.46.25.66 with SMTP id p63mr458721lje.12.1506342681389; Mon, 25 Sep 2017 05:31:21 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.174.129 with SMTP id x123ls1194380wme.3.gmail; Mon, 25 Sep 2017 05:31:21 -0700 (PDT) X-Google-Smtp-Source: AOwi7QAuHy/RJfIzzh6uZf33WsIodKGOAg5d0A25tk0swQU4cf7TN5RIEvTx2nb4+7/RfQ+S4kfH X-Received: by 10.28.134.137 with SMTP id i131mr31804wmd.29.1506342681034; Mon, 25 Sep 2017 05:31:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1506342680; cv=none; d=google.com; s=arc-20160816; b=WmuIW+rUcuPtvLTbCKyLUHlswQdwSeR2iVA2BaZeskm28ldDCmpvps9RS0sApOcB9P kLxb0Y2CF5ufjUS98dQ2ZgX7vysTrLh4cDJwRtGBN2SMVQPBBx0D3y5K/3GX6MuqXfki TjATVu1j9cgRteH5h0QiodeXJUHCF5oW4ZGSTpbztJuN1USrTO+LHhPNGZGrh3P9JbCN b6sXlI1mRkhKigfkCX9G/LLfjDst7eib9FyourMep9ZWY+UuAHgZdgzBZ784i4XxdZbB 5xxHfitFHptgfgac6tgI8+I015bwlhpt6fdv7Sb2DU4QWszqVDAy/Bf8JJnDXR6nSe/a tjPw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject :arc-authentication-results; bh=17rHnBMN519QK3Xiqi4y72MiJrSNact+stfvsqW3K9c=; b=zUCCtEvFfrBZIJpkahxudPQ4K1MUUQRZpqQSYgpKG4yo+9FgM3E+C8+nZsEhH/gwLE L1b5KJ2B43NfO6IqbjZzBN3J/jceNHLry7MTnDbe3qHZQXMm8oYoZfrD3t3IDGW4YCIh fQeV2j1lUpQdp1MqaeUgWvHtN/dNO4XG+J3oKtYpiugWKbk0V772O8wX3uPx9QYzYd4p YQeQiOyqgusdZqGM72PzB0NvgffdqTTZTcXLwtVP9XXTVdCOrTq/2xQCL8mZkD6dl/IH gdI83SDkJ5jPZihYPpRFolGZeikeN5SJwnhgtJBVRnRUUFzeW4LdYVrLMYKo3y9HIsCj ZaSA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id p200si472810wmg.4.2017.09.25.05.31.20 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 25 Sep 2017 05:31:20 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Received: from [10.0.2.15] ([188.227.110.165]) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id v8PCVIsG002471 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Mon, 25 Sep 2017 14:31:19 +0200 Subject: Re: [PATCH v3 2/4] meta-isar-bin: Generate cache repos To: Henning Schild Cc: isar-users@googlegroups.com References: <20170925100000.5368-1-asmirnov@ilbers.de> <20170925100000.5368-3-asmirnov@ilbers.de> <20170925134355.6ec87848@md1em3qc> From: Alexander Smirnov Message-ID: <4be5e9c9-ee92-b18f-030b-cab3f8c5e91e@ilbers.de> Date: Mon, 25 Sep 2017 15:31:13 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: <20170925134355.6ec87848@md1em3qc> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: tdH0pFHPUiha On 09/25/2017 02:43 PM, Henning Schild wrote: > Am Mon, 25 Sep 2017 12:59:58 +0300 > schrieb Alexander Smirnov : > >> Generate repos for each configuration in multiconfig. >> >> Signed-off-by: Alexander Smirnov >> --- >> meta-isar-bin/conf/layer.conf | 3 +++ >> meta-isar/conf/local.conf.sample | 4 ++++ >> meta/classes/image.bbclass | 16 ++++++++++++++++ >> 3 files changed, 23 insertions(+) >> >> diff --git a/meta-isar-bin/conf/layer.conf >> b/meta-isar-bin/conf/layer.conf index 3518184..5ef6e60 100644 >> --- a/meta-isar-bin/conf/layer.conf >> +++ b/meta-isar-bin/conf/layer.conf >> @@ -9,3 +9,6 @@ DEBCACHEDIR ?= "${LAYERDIR}/apt" >> >> # Path to the configuration files templates used by `reprepro` >> DEBFILESDIR ?= "${LAYERDIR}/files" >> + >> +# Path to the databases used by `reprepro` >> +DEBDBDIR ?= "${LAYERDIR}/db" >> diff --git a/meta-isar/conf/local.conf.sample >> b/meta-isar/conf/local.conf.sample index a456b1b..660958f 100644 >> --- a/meta-isar/conf/local.conf.sample >> +++ b/meta-isar/conf/local.conf.sample >> @@ -158,3 +158,7 @@ IMAGE_INSTALL = "hello example-raw" >> # >> # Default parallel jobs for bitbake: >> BB_NUMBER_THREADS = "4" >> + >> +# >> +# Number of attempts to try to get reprepro lock for access to apt >> cache +REPREPRO_LOCK_ATTEMPTS = "16" >> >> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass >> index 9ab9b19..6f39c7a 100644 >> --- a/meta/classes/image.bbclass >> +++ b/meta/classes/image.bbclass >> @@ -14,11 +14,27 @@ CACHE_CONF_DIR = "${DEBCACHEDIR}/${DISTRO}/conf" >> do_cache_config[dirs] = "${CACHE_CONF_DIR}" >> do_cache_config[stamp-extra-info] = "${DISTRO}" >> >> +call_reprepro() { >> + for i in $(seq 1 ${REPREPRO_LOCK_ATTEMPTS}); do >> + reprepro --waitforlock 5 $* \ >> + || bbwarn "Failed to get reprepro lock, trying again..." > > Now we have two loops and the number from above need to be bultiplied > with the inner number .. 5*16 times 10s. I still do not see how that > solves the problem. But i can imagine those two loops really slowing > down builds that fail. Looks like they potentially add 800s build-time. > And every error of reprepro is turned into "Failed to get lock" ... As > far as i looked at reprepro it should return EEXIST in the > contention case. > This is the compromise to avoid races in reprepro. Some thoughts: 1. Infinite loop is not an option, because I expect that broken build will eventually exit. 2. I want to have some notifications if lock acquisition fails for a long time. 3. Possible number of races depends on amount of packages to be processed. So this variable should be tuned anyway. Until reprepro can't query processing from different threads and bitbake doesn't provide lock primitives, I don't see how this could be solved permanently at the moment. I've checked reprepro return code: $ reprepro -b meta-isar-bin/apt/debian-stretch/ --dbdir ./meta-isar-bin/db/debian-stretch/ -C main ls hello || echo $? ... 239 So looks like the code is really EEXIST. In this case I'd like to add check for this exitcode and keep the loop as it is. Any other suggestions? Alex > Henning > >> + done >> +} >> + >> do_cache_config() { >> if [ ! -e "${CACHE_CONF_DIR}/distributions" ]; then >> sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \ >> ${DEBFILESDIR}/distributions.in > >> ${CACHE_CONF_DIR}/distributions fi >> + >> + path_cache="${DEBCACHEDIR}/${DISTRO}" >> + path_databases="${DEBDBDIR}/${DISTRO}" >> + >> + if [ ! -d "${path_databases}" ]; then >> + call_reprepro -b ${path_cache} \ >> + --dbdir ${path_databases} \ >> + export ${DEBDISTRONAME} >> + fi >> } >> >> addtask cache_config before do_fetch > -- With best regards, Alexander Smirnov ilbers GmbH Baierbrunner Str. 28c D-81379 Munich +49 (89) 122 67 24-0 http://ilbers.de/ Commercial register Munich, HRB 214197 General manager: Baurzhan Ismagulov