Re: [PATCH v2 3/4] meta-isar-bin: Populate cache

[Alexander Smirnov <asmirnov@ilbers.de>]

On 09/20/2017 11:22 AM, Henning Schild wrote:
> Am Tue, 19 Sep 2017 15:20:51 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> Add newly built packages to apt cache.
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>>   meta/classes/image.bbclass | 9 +++++++++
>>   1 file changed, 9 insertions(+)
>>
>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
>> index 9db7371..d59959b 100644
>> --- a/meta/classes/image.bbclass
>> +++ b/meta/classes/image.bbclass
>> @@ -54,6 +54,15 @@ do_populate[stamp-extra-info] =
>> "${DISTRO}-${MACHINE}" # Install Debian packages, that were built
>> from sources do_populate() {
>>       if [ -n "${IMAGE_INSTALL}" ]; then
>> +        for p in ${IMAGE_INSTALL}; do
>> +            reprepro --waitforlock 3 \
>> +                     -b ${DEBCACHEDIR}/${DISTRO} \
>> +                     --dbdir ${DEBDBDIR}/${DISTRO} \
>> +                     -C main \
>> +                     includedeb ${DEBDISTRONAME} \
>> +                     ${DEPLOY_DIR_DEB}/${p}_*.deb
>> +        done
> 
> Why not just do that in do_deploy_deb? Maybe DEPLOY_DIR_DEB could go
> away because we deploy directly into the repo.

This is an attempt to avoid races in reprepro, let me describe the usecase:

1. do_populate() is executed after do_build()

2. If we have several recipes for building, this means that 
do_populate() could be run in parallel (if you enable parallel jobs for 
bitbake)

3. --waitforlock does not guarantee that reprepro will take ownership 
after timeout

4. I agree that DEPLOY_DIR_DEB should go away, but at the moment I think 
we can use it for a while, until better solution will be found.

One approach could be walking through WORKDIRS:

for p in ${IMAGE_INSTALL}; do
     reprepro \
     ....
     ${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${p}-*/*.deb
done

but I'd like to think if some better solution could be found.

> 
> I found that for kernels built with "make deb-pkg" the "${p}_*.deb"
> pattern does not match. At least the way that i named the recipe.
> "do_deploy_deb" already works on WORKDIR/*.deb which naturally covers
> all packages coming out of a recipe, even if they had names different
> from the recipe. Imagine a recipe that downloads multiple .debs from
> somewhere and they all have very different names.

This works with ${WORKDIR}, where you know that is contains only 
packages you want (i.e. they are listed in ${IMAGE_INSTALL}). But deploy 
directory could contain 'unwanted' packages, that for example are left 
from previous builds. But I think we could ignore this case for now.

> 
> Again the question about the waitforlock, 3 seems only slightly better
> than 1. I think we need infinite or a serialization mechanism and 1.

Hmm, here you are right. If several architectures in multiconfig will 
populate ${DEPLOY_DIR_DEB} in parallel, 3 times won't be enough. Will 
try to implement loop to poll return value from reprepro without 
waitforlock.

Alex

> 
> Henning
> 
>>           sudo mkdir -p ${IMAGE_ROOTFS}/deb
>>   
>>           for p in ${IMAGE_INSTALL}; do
>