Re: [PATCH v2 1/4] meta-isar-bin: Add reprepro configs

[Alexander Smirnov <asmirnov@ilbers.de>]

On 09/20/2017 11:38 AM, Henning Schild wrote:
> Am Wed, 20 Sep 2017 11:12:11 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> On 09/20/2017 10:58 AM, Henning Schild wrote:
>>> Am Tue, 19 Sep 2017 15:20:49 +0300
>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>    
>>>> Add template for reprepro configuration file and function that
>>>> generates final config for specific deistro.
>>>>
>>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>>>> ---
>>>>    meta-isar-bin/conf/layer.conf        | 11 +++++++++++
>>>>    meta-isar-bin/files/distributions.in |  3 +++
>>>>    meta-isar/conf/bblayers.conf.sample  |  1 +
>>>>    meta/classes/image.bbclass           | 25
>>>> +++++++++++++++++++++++++ 4 files changed, 40 insertions(+)
>>>>    create mode 100644 meta-isar-bin/conf/layer.conf
>>>>    create mode 100644 meta-isar-bin/files/distributions.in
>>>>
>>>> diff --git a/meta-isar-bin/conf/layer.conf
>>>> b/meta-isar-bin/conf/layer.conf new file mode 100644
>>>> index 0000000..3518184
>>>> --- /dev/null
>>>> +++ b/meta-isar-bin/conf/layer.conf
>>>> @@ -0,0 +1,11 @@
>>>> +# This software is a part of ISAR.
>>>> +# Copyright (C) 2017 ilbers GmbH
>>>> +
>>>> +# Codename of the repository created by the caching class
>>>> +DEBDISTRONAME = "isar"
>>>> +
>>>> +# Path to the caching repository
>>>> +DEBCACHEDIR ?= "${LAYERDIR}/apt"
>>>> +
>>>> +# Path to the configuration files templates used by `reprepro`
>>>> +DEBFILESDIR ?= "${LAYERDIR}/files"
>>>> diff --git a/meta-isar-bin/files/distributions.in
>>>> b/meta-isar-bin/files/distributions.in new file mode 100644
>>>> index 0000000..cd214c6
>>>> --- /dev/null
>>>> +++ b/meta-isar-bin/files/distributions.in
>>>> @@ -0,0 +1,3 @@
>>>> +Codename: {DISTRO_NAME}
>>>> +Architectures: i386 armhf amd64 source
>>>> +Components: main
>>>> diff --git a/meta-isar/conf/bblayers.conf.sample
>>>> b/meta-isar/conf/bblayers.conf.sample index 80867e7..53a362b 100644
>>>> --- a/meta-isar/conf/bblayers.conf.sample
>>>> +++ b/meta-isar/conf/bblayers.conf.sample
>>>> @@ -8,6 +8,7 @@ BBFILES ?= ""
>>>>    BBLAYERS ?= " \
>>>>      ##ISARROOT##/meta \
>>>>      ##ISARROOT##/meta-isar \
>>>> +  ##ISARROOT##/meta-isar-bin \
>>>>      "
>>>>    BBLAYERS_NON_REMOVABLE ?= " \
>>>>      ##ISARROOT##/meta \
>>>> diff --git a/meta/classes/image.bbclass
>>>> b/meta/classes/image.bbclass index 5bf9524..30f241a 100644
>>>> --- a/meta/classes/image.bbclass
>>>> +++ b/meta/classes/image.bbclass
>>>> @@ -10,6 +10,31 @@ IMAGE_ROOTFS   = "${WORKDIR}/rootfs"
>>>>    
>>>>    inherit ${IMAGE_TYPE}
>>>>    
>>>> +# Generate apt cache configuration file
>>>> +python __anonymous () {
>>>> +    DISTRO = d.getVar("DISTRO", True)
>>>> +    DEBFILESDIR = d.getVar("DEBFILESDIR", True)
>>>> +    DEBCACHEDIR = d.getVar("DEBCACHEDIR", True)
>>>> +
>>>> +    cache_conf_path = os.path.join(DEBCACHEDIR, DISTRO, "conf")
>>>> +    cache_conf_file = os.path.join(cache_conf_path,
>>>> "distributions")
>>>> +    cache_conf_file_in = os.path.join(DEBFILESDIR,
>>>> "distributions.in") +
>>>> +    import subprocess
>>>> +
>>>> +    if not os.path.exists(cache_conf_path):
>>>> +        os.makedirs(cache_conf_path)
>>>> +
>>>> +    if not os.path.exists(cache_conf_file):
>>>> +        f = open(cache_conf_file, "w")
>>>> +        subprocess.check_call([
>>>> +            "sed",
>>>> +            "-e", "s#{DISTRO_NAME}#" + d.getVar("DEBDISTRONAME",
>>>> True) + "#g",
>>>> +            cache_conf_file_in,
>>>> +            ], stdout=f)
>>>> +        f.close()
>>>> +}
>>>
>>> Why is this not just a regular task early in the image chain?
>>>    
>>
>> The idea is to avoid races with multiconfig. If I add this as a
>> regular task, it could be run in parallel by different multiconfigs,
>> so there could occur races in `mkdir` and `sed`. The same is true
>> when you try to build several images at the same time:
>>
>>     $ bitbake image1 image2
>>
>> IIRC bitbake anonymous tasks are executed during parsing stage, so
>> they are serialized.
> 
> I see, but that makes them a very strong measure that you can never
> override i.e. in a layer on top. Not sure if that matters but i would
> shy away from using the strongest measure if it can be done in a
> different way.

Agree.

> 
> For the dir how about using newtask[dirs] i guess bitbake will know how
> to handle that and it seems to be the bitbake way anyways.

Let me try :-)

> 
> And for the sed, if the race happens and two or more seds get run. They
> should just produce the same output and not fail if it was already there
> 
> sed "s#{DISTRO_NAME}#" + d.getVar("DEBDISTRONAME", True) + "#g" \
> 	cache_conf_file_in > cache_conf_file
> 

Ok.

> Henning
> 
>> Alex
>>
>>> Henning
>>>    
>>>>    do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}"
>>>>    
>>>>    # Install Debian packages, that were built from sources
>>>   
>