Re: [PATCH v3 2/4] meta-isar-bin: Generate cache repos

[Henning Schild <henning.schild@siemens.com>]

Am Mon, 25 Sep 2017 15:31:13 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> On 09/25/2017 02:43 PM, Henning Schild wrote:
> > Am Mon, 25 Sep 2017 12:59:58 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >   
> >> Generate repos for each configuration in multiconfig.
> >>
> >> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> >> ---
> >>   meta-isar-bin/conf/layer.conf    |  3 +++
> >>   meta-isar/conf/local.conf.sample |  4 ++++
> >>   meta/classes/image.bbclass       | 16 ++++++++++++++++
> >>   3 files changed, 23 insertions(+)
> >>
> >> diff --git a/meta-isar-bin/conf/layer.conf
> >> b/meta-isar-bin/conf/layer.conf index 3518184..5ef6e60 100644
> >> --- a/meta-isar-bin/conf/layer.conf
> >> +++ b/meta-isar-bin/conf/layer.conf
> >> @@ -9,3 +9,6 @@ DEBCACHEDIR ?= "${LAYERDIR}/apt"
> >>   
> >>   # Path to the configuration files templates used by `reprepro`
> >>   DEBFILESDIR ?= "${LAYERDIR}/files"
> >> +
> >> +# Path to the databases used by `reprepro`
> >> +DEBDBDIR ?= "${LAYERDIR}/db"
> >> diff --git a/meta-isar/conf/local.conf.sample
> >> b/meta-isar/conf/local.conf.sample index a456b1b..660958f 100644
> >> --- a/meta-isar/conf/local.conf.sample
> >> +++ b/meta-isar/conf/local.conf.sample
> >> @@ -158,3 +158,7 @@ IMAGE_INSTALL = "hello example-raw"
> >>   #
> >>   # Default parallel jobs for bitbake:
> >>   BB_NUMBER_THREADS = "4"
> >> +
> >> +#
> >> +# Number of attempts to try to get reprepro lock for access to apt
> >> cache +REPREPRO_LOCK_ATTEMPTS = "16"
> >>
> >> diff --git a/meta/classes/image.bbclass
> >> b/meta/classes/image.bbclass index 9ab9b19..6f39c7a 100644
> >> --- a/meta/classes/image.bbclass
> >> +++ b/meta/classes/image.bbclass
> >> @@ -14,11 +14,27 @@ CACHE_CONF_DIR =
> >> "${DEBCACHEDIR}/${DISTRO}/conf" do_cache_config[dirs] =
> >> "${CACHE_CONF_DIR}" do_cache_config[stamp-extra-info] = "${DISTRO}"
> >>   
> >> +call_reprepro() {
> >> +    for i in $(seq 1 ${REPREPRO_LOCK_ATTEMPTS}); do
> >> +        reprepro --waitforlock 5 $* \
> >> +        || bbwarn "Failed to get reprepro lock, trying again..."  
> > 
> > Now we have two loops and the number from above need to be
> > bultiplied with the inner number .. 5*16 times 10s. I still do not
> > see how that solves the problem. But i can imagine those two loops
> > really slowing down builds that fail. Looks like they potentially
> > add 800s build-time. And every error of reprepro is turned into
> > "Failed to get lock" ... As far as i looked at reprepro it should
> > return EEXIST in the contention case.
> >   
> 
> This is the compromise to avoid races in reprepro. Some thoughts:
> 
> 1. Infinite loop is not an option, because I expect that broken build 
> will eventually exit.
> 
> 2. I want to have some notifications if lock acquisition fails for a 
> long time.
> 
> 3. Possible number of races depends on amount of packages to be 
> processed. So this variable should be tuned anyway.
> 
> Until reprepro can't query processing from different threads and
> bitbake doesn't provide lock primitives, I don't see how this could
> be solved permanently at the moment.
> 
> I've checked reprepro return code:
> 
> $ reprepro -b meta-isar-bin/apt/debian-stretch/ --dbdir 
> ./meta-isar-bin/db/debian-stretch/ -C main ls hello || echo $?
> ...
> 239

EEXIST should be 17 and this looks like a uint8 -EEXIST, best use
python for getting the number
import errno; print(errno.EEXIST)

> So looks like the code is really EEXIST. In this case I'd like to add 
> check for this exitcode and keep the loop as it is. Any other
> suggestions?

Yes, the inner loop should use a "--waitforlock 1" instead of 5.

Henning

> Alex
> 
> > Henning
> >   
> >> +    done
> >> +}
> >> +
> >>   do_cache_config() {
> >>       if [ ! -e "${CACHE_CONF_DIR}/distributions" ]; then
> >>           sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \
> >>               ${DEBFILESDIR}/distributions.in >
> >> ${CACHE_CONF_DIR}/distributions fi
> >> +
> >> +    path_cache="${DEBCACHEDIR}/${DISTRO}"
> >> +    path_databases="${DEBDBDIR}/${DISTRO}"
> >> +
> >> +    if [ ! -d "${path_databases}" ]; then
> >> +        call_reprepro -b ${path_cache} \
> >> +                      --dbdir ${path_databases} \
> >> +                      export ${DEBDISTRONAME}
> >> +    fi
> >>   }
> >>   
> >>   addtask cache_config before do_fetch  
> >   
>