From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6467463440282681344 X-Received: by 10.28.145.82 with SMTP id t79mr57849wmd.20.1506347540765; Mon, 25 Sep 2017 06:52:20 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.191.29 with SMTP id p29ls1233340wmf.8.canary-gmail; Mon, 25 Sep 2017 06:52:20 -0700 (PDT) X-Google-Smtp-Source: AOwi7QBCG6e+kKTpA3DSLMM5cVlkM0rIJXOfwFG1rmiPc/pAWb74cPPthh5Le0Uk5gOcbaavfkkK X-Received: by 10.28.230.79 with SMTP id d76mr55382wmh.12.1506347540312; Mon, 25 Sep 2017 06:52:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1506347539; cv=none; d=google.com; s=arc-20160816; b=F8S5L/f5x7BkvTUbx2FLA9YEZNcyG3T8LC6CRZ4BDcyZ9d7zLnRoO79jQJpGUeUABS IYVGOcATAULopERxYdvHlI8reKCcCcsMSm9QXYTFhgHZrXgmr3agsJhUfQ8HfLp/Jfl3 Sl5htwkUQ31RyHVGM+toR6wJxsTM3oINaVNBa/PEu8ZwVM9i+El8M9pin+8esOHIwiJM 73GXWAxd8XKchkwJbq2j5fliXP3WtTmGvBsVqWEjiBXDtWb51kclfTSqHIeKuN0bkCol b8RGZpziI0BaJuoxarzhakUc++cSHIipXtPw36/hCj4YCAh3O1cSxNeZPkCbv3Lt+TBO j5Hg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:arc-authentication-results; bh=iW6i11vaeRUTOfhsTIEtyK7FGTPmwHOItIWaJ+LJ8aw=; b=sEY6h0vXhV+t4mNrAqWBTENA/5YEivAS8nPkvyzw78VlGs9yTjqbMLCwwrMYI7i5eK bmfV9AbbaaFF69cvOYUZdYRNOiMePmiQT2JW1FxT7EaGKeeCeB5t/aLh/cC7K1OC6OPc rAGSFtSWpG6NDdjyMJV5H7JzeAQDoZYh7rs/dGtk4quEi84/vNw6TMgdTFz0HhTspIND LBNcYJP5QVm0j/JFNAnaToC0jyzvfvrL1O/5ngLxvHxiK+zZTVSynZ3Y8I9/WNh0Owfh 5iC6ViQauyWfUNomdzxMiyrOP8BTZOUEDrhOLUNkDbp0FJIQmS+yqFE1NCHP+rEYzhj5 oysA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 192.35.17.14 is neither permitted nor denied by domain of henning.schild@siemens.com) smtp.mailfrom=henning.schild@siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id b130si425538wme.0.2017.09.25.06.52.19 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 25 Sep 2017 06:52:19 -0700 (PDT) Received-SPF: neutral (google.com: 192.35.17.14 is neither permitted nor denied by domain of henning.schild@siemens.com) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 192.35.17.14 is neither permitted nor denied by domain of henning.schild@siemens.com) smtp.mailfrom=henning.schild@siemens.com Received: from mail3.siemens.de (mail3.siemens.de [139.25.208.14]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id v8PDqJFT011205 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 25 Sep 2017 15:52:19 +0200 Received: from md1em3qc ([139.25.68.40]) by mail3.siemens.de (8.15.2/8.15.2) with ESMTP id v8PDqJ1u007651; Mon, 25 Sep 2017 15:52:19 +0200 Date: Mon, 25 Sep 2017 15:52:38 +0200 From: Henning Schild To: Alexander Smirnov Cc: Subject: Re: [PATCH v3 2/4] meta-isar-bin: Generate cache repos Message-ID: <20170925155238.215b334d@md1em3qc> In-Reply-To: <4be5e9c9-ee92-b18f-030b-cab3f8c5e91e@ilbers.de> References: <20170925100000.5368-1-asmirnov@ilbers.de> <20170925100000.5368-3-asmirnov@ilbers.de> <20170925134355.6ec87848@md1em3qc> <4be5e9c9-ee92-b18f-030b-cab3f8c5e91e@ilbers.de> X-Mailer: Claws Mail 3.13.2 (GTK+ 2.24.31; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-TUID: 4IkX4SxgOA7K Am Mon, 25 Sep 2017 15:31:13 +0300 schrieb Alexander Smirnov : > On 09/25/2017 02:43 PM, Henning Schild wrote: > > Am Mon, 25 Sep 2017 12:59:58 +0300 > > schrieb Alexander Smirnov : > > > >> Generate repos for each configuration in multiconfig. > >> > >> Signed-off-by: Alexander Smirnov > >> --- > >> meta-isar-bin/conf/layer.conf | 3 +++ > >> meta-isar/conf/local.conf.sample | 4 ++++ > >> meta/classes/image.bbclass | 16 ++++++++++++++++ > >> 3 files changed, 23 insertions(+) > >> > >> diff --git a/meta-isar-bin/conf/layer.conf > >> b/meta-isar-bin/conf/layer.conf index 3518184..5ef6e60 100644 > >> --- a/meta-isar-bin/conf/layer.conf > >> +++ b/meta-isar-bin/conf/layer.conf > >> @@ -9,3 +9,6 @@ DEBCACHEDIR ?= "${LAYERDIR}/apt" > >> > >> # Path to the configuration files templates used by `reprepro` > >> DEBFILESDIR ?= "${LAYERDIR}/files" > >> + > >> +# Path to the databases used by `reprepro` > >> +DEBDBDIR ?= "${LAYERDIR}/db" > >> diff --git a/meta-isar/conf/local.conf.sample > >> b/meta-isar/conf/local.conf.sample index a456b1b..660958f 100644 > >> --- a/meta-isar/conf/local.conf.sample > >> +++ b/meta-isar/conf/local.conf.sample > >> @@ -158,3 +158,7 @@ IMAGE_INSTALL = "hello example-raw" > >> # > >> # Default parallel jobs for bitbake: > >> BB_NUMBER_THREADS = "4" > >> + > >> +# > >> +# Number of attempts to try to get reprepro lock for access to apt > >> cache +REPREPRO_LOCK_ATTEMPTS = "16" > >> > >> diff --git a/meta/classes/image.bbclass > >> b/meta/classes/image.bbclass index 9ab9b19..6f39c7a 100644 > >> --- a/meta/classes/image.bbclass > >> +++ b/meta/classes/image.bbclass > >> @@ -14,11 +14,27 @@ CACHE_CONF_DIR = > >> "${DEBCACHEDIR}/${DISTRO}/conf" do_cache_config[dirs] = > >> "${CACHE_CONF_DIR}" do_cache_config[stamp-extra-info] = "${DISTRO}" > >> > >> +call_reprepro() { > >> + for i in $(seq 1 ${REPREPRO_LOCK_ATTEMPTS}); do > >> + reprepro --waitforlock 5 $* \ > >> + || bbwarn "Failed to get reprepro lock, trying again..." > > > > Now we have two loops and the number from above need to be > > bultiplied with the inner number .. 5*16 times 10s. I still do not > > see how that solves the problem. But i can imagine those two loops > > really slowing down builds that fail. Looks like they potentially > > add 800s build-time. And every error of reprepro is turned into > > "Failed to get lock" ... As far as i looked at reprepro it should > > return EEXIST in the contention case. > > > > This is the compromise to avoid races in reprepro. Some thoughts: > > 1. Infinite loop is not an option, because I expect that broken build > will eventually exit. > > 2. I want to have some notifications if lock acquisition fails for a > long time. > > 3. Possible number of races depends on amount of packages to be > processed. So this variable should be tuned anyway. > > Until reprepro can't query processing from different threads and > bitbake doesn't provide lock primitives, I don't see how this could > be solved permanently at the moment. > > I've checked reprepro return code: > > $ reprepro -b meta-isar-bin/apt/debian-stretch/ --dbdir > ./meta-isar-bin/db/debian-stretch/ -C main ls hello || echo $? > ... > 239 EEXIST should be 17 and this looks like a uint8 -EEXIST, best use python for getting the number import errno; print(errno.EEXIST) > So looks like the code is really EEXIST. In this case I'd like to add > check for this exitcode and keep the loop as it is. Any other > suggestions? Yes, the inner loop should use a "--waitforlock 1" instead of 5. Henning > Alex > > > Henning > > > >> + done > >> +} > >> + > >> do_cache_config() { > >> if [ ! -e "${CACHE_CONF_DIR}/distributions" ]; then > >> sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \ > >> ${DEBFILESDIR}/distributions.in > > >> ${CACHE_CONF_DIR}/distributions fi > >> + > >> + path_cache="${DEBCACHEDIR}/${DISTRO}" > >> + path_databases="${DEBDBDIR}/${DISTRO}" > >> + > >> + if [ ! -d "${path_databases}" ]; then > >> + call_reprepro -b ${path_cache} \ > >> + --dbdir ${path_databases} \ > >> + export ${DEBDISTRONAME} > >> + fi > >> } > >> > >> addtask cache_config before do_fetch > > >