Re: [PATCH v3 2/4] meta-isar-bin: Generate cache repos

[Alexander Smirnov <asmirnov@ilbers.de>]

On 09/25/2017 02:43 PM, Henning Schild wrote:
> Am Mon, 25 Sep 2017 12:59:58 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> Generate repos for each configuration in multiconfig.
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>>   meta-isar-bin/conf/layer.conf    |  3 +++
>>   meta-isar/conf/local.conf.sample |  4 ++++
>>   meta/classes/image.bbclass       | 16 ++++++++++++++++
>>   3 files changed, 23 insertions(+)
>>
>> diff --git a/meta-isar-bin/conf/layer.conf
>> b/meta-isar-bin/conf/layer.conf index 3518184..5ef6e60 100644
>> --- a/meta-isar-bin/conf/layer.conf
>> +++ b/meta-isar-bin/conf/layer.conf
>> @@ -9,3 +9,6 @@ DEBCACHEDIR ?= "${LAYERDIR}/apt"
>>   
>>   # Path to the configuration files templates used by `reprepro`
>>   DEBFILESDIR ?= "${LAYERDIR}/files"
>> +
>> +# Path to the databases used by `reprepro`
>> +DEBDBDIR ?= "${LAYERDIR}/db"
>> diff --git a/meta-isar/conf/local.conf.sample
>> b/meta-isar/conf/local.conf.sample index a456b1b..660958f 100644
>> --- a/meta-isar/conf/local.conf.sample
>> +++ b/meta-isar/conf/local.conf.sample
>> @@ -158,3 +158,7 @@ IMAGE_INSTALL = "hello example-raw"
>>   #
>>   # Default parallel jobs for bitbake:
>>   BB_NUMBER_THREADS = "4"
>> +
>> +#
>> +# Number of attempts to try to get reprepro lock for access to apt
>> cache +REPREPRO_LOCK_ATTEMPTS = "16"
>>
>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
>> index 9ab9b19..6f39c7a 100644
>> --- a/meta/classes/image.bbclass
>> +++ b/meta/classes/image.bbclass
>> @@ -14,11 +14,27 @@ CACHE_CONF_DIR = "${DEBCACHEDIR}/${DISTRO}/conf"
>>   do_cache_config[dirs] = "${CACHE_CONF_DIR}"
>>   do_cache_config[stamp-extra-info] = "${DISTRO}"
>>   
>> +call_reprepro() {
>> +    for i in $(seq 1 ${REPREPRO_LOCK_ATTEMPTS}); do
>> +        reprepro --waitforlock 5 $* \
>> +        || bbwarn "Failed to get reprepro lock, trying again..."
> 
> Now we have two loops and the number from above need to be bultiplied
> with the inner number .. 5*16 times 10s. I still do not see how that
> solves the problem. But i can imagine those two loops really slowing
> down builds that fail. Looks like they potentially add 800s build-time.
> And every error of reprepro is turned into "Failed to get lock" ... As
> far as i looked at reprepro it should return EEXIST in the
> contention case.
> 

This is the compromise to avoid races in reprepro. Some thoughts:

1. Infinite loop is not an option, because I expect that broken build 
will eventually exit.

2. I want to have some notifications if lock acquisition fails for a 
long time.

3. Possible number of races depends on amount of packages to be 
processed. So this variable should be tuned anyway.

Until reprepro can't query processing from different threads and bitbake 
doesn't provide lock primitives, I don't see how this could be solved 
permanently at the moment.

I've checked reprepro return code:

$ reprepro -b meta-isar-bin/apt/debian-stretch/ --dbdir 
./meta-isar-bin/db/debian-stretch/ -C main ls hello || echo $?
...
239

So looks like the code is really EEXIST. In this case I'd like to add 
check for this exitcode and keep the loop as it is. Any other suggestions?

Alex

> Henning
> 
>> +    done
>> +}
>> +
>>   do_cache_config() {
>>       if [ ! -e "${CACHE_CONF_DIR}/distributions" ]; then
>>           sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \
>>               ${DEBFILESDIR}/distributions.in >
>> ${CACHE_CONF_DIR}/distributions fi
>> +
>> +    path_cache="${DEBCACHEDIR}/${DISTRO}"
>> +    path_databases="${DEBDBDIR}/${DISTRO}"
>> +
>> +    if [ ! -d "${path_databases}" ]; then
>> +        call_reprepro -b ${path_cache} \
>> +                      --dbdir ${path_databases} \
>> +                      export ${DEBDISTRONAME}
>> +    fi
>>   }
>>   
>>   addtask cache_config before do_fetch
> 

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov