Re: [PATCH v2 1/4] meta-isar-bin: Add reprepro configs

[Henning Schild <henning.schild@siemens.com>]

Am Wed, 20 Sep 2017 11:12:11 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> On 09/20/2017 10:58 AM, Henning Schild wrote:
> > Am Tue, 19 Sep 2017 15:20:49 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >   
> >> Add template for reprepro configuration file and function that
> >> generates final config for specific deistro.
> >>
> >> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> >> ---
> >>   meta-isar-bin/conf/layer.conf        | 11 +++++++++++
> >>   meta-isar-bin/files/distributions.in |  3 +++
> >>   meta-isar/conf/bblayers.conf.sample  |  1 +
> >>   meta/classes/image.bbclass           | 25
> >> +++++++++++++++++++++++++ 4 files changed, 40 insertions(+)
> >>   create mode 100644 meta-isar-bin/conf/layer.conf
> >>   create mode 100644 meta-isar-bin/files/distributions.in
> >>
> >> diff --git a/meta-isar-bin/conf/layer.conf
> >> b/meta-isar-bin/conf/layer.conf new file mode 100644
> >> index 0000000..3518184
> >> --- /dev/null
> >> +++ b/meta-isar-bin/conf/layer.conf
> >> @@ -0,0 +1,11 @@
> >> +# This software is a part of ISAR.
> >> +# Copyright (C) 2017 ilbers GmbH
> >> +
> >> +# Codename of the repository created by the caching class
> >> +DEBDISTRONAME = "isar"
> >> +
> >> +# Path to the caching repository
> >> +DEBCACHEDIR ?= "${LAYERDIR}/apt"
> >> +
> >> +# Path to the configuration files templates used by `reprepro`
> >> +DEBFILESDIR ?= "${LAYERDIR}/files"
> >> diff --git a/meta-isar-bin/files/distributions.in
> >> b/meta-isar-bin/files/distributions.in new file mode 100644
> >> index 0000000..cd214c6
> >> --- /dev/null
> >> +++ b/meta-isar-bin/files/distributions.in
> >> @@ -0,0 +1,3 @@
> >> +Codename: {DISTRO_NAME}
> >> +Architectures: i386 armhf amd64 source
> >> +Components: main
> >> diff --git a/meta-isar/conf/bblayers.conf.sample
> >> b/meta-isar/conf/bblayers.conf.sample index 80867e7..53a362b 100644
> >> --- a/meta-isar/conf/bblayers.conf.sample
> >> +++ b/meta-isar/conf/bblayers.conf.sample
> >> @@ -8,6 +8,7 @@ BBFILES ?= ""
> >>   BBLAYERS ?= " \
> >>     ##ISARROOT##/meta \
> >>     ##ISARROOT##/meta-isar \
> >> +  ##ISARROOT##/meta-isar-bin \
> >>     "
> >>   BBLAYERS_NON_REMOVABLE ?= " \
> >>     ##ISARROOT##/meta \
> >> diff --git a/meta/classes/image.bbclass
> >> b/meta/classes/image.bbclass index 5bf9524..30f241a 100644
> >> --- a/meta/classes/image.bbclass
> >> +++ b/meta/classes/image.bbclass
> >> @@ -10,6 +10,31 @@ IMAGE_ROOTFS   = "${WORKDIR}/rootfs"
> >>   
> >>   inherit ${IMAGE_TYPE}
> >>   
> >> +# Generate apt cache configuration file
> >> +python __anonymous () {
> >> +    DISTRO = d.getVar("DISTRO", True)
> >> +    DEBFILESDIR = d.getVar("DEBFILESDIR", True)
> >> +    DEBCACHEDIR = d.getVar("DEBCACHEDIR", True)
> >> +
> >> +    cache_conf_path = os.path.join(DEBCACHEDIR, DISTRO, "conf")
> >> +    cache_conf_file = os.path.join(cache_conf_path,
> >> "distributions")
> >> +    cache_conf_file_in = os.path.join(DEBFILESDIR,
> >> "distributions.in") +
> >> +    import subprocess
> >> +
> >> +    if not os.path.exists(cache_conf_path):
> >> +        os.makedirs(cache_conf_path)
> >> +
> >> +    if not os.path.exists(cache_conf_file):
> >> +        f = open(cache_conf_file, "w")
> >> +        subprocess.check_call([
> >> +            "sed",
> >> +            "-e", "s#{DISTRO_NAME}#" + d.getVar("DEBDISTRONAME",
> >> True) + "#g",
> >> +            cache_conf_file_in,
> >> +            ], stdout=f)
> >> +        f.close()
> >> +}  
> > 
> > Why is this not just a regular task early in the image chain?
> >   
> 
> The idea is to avoid races with multiconfig. If I add this as a
> regular task, it could be run in parallel by different multiconfigs,
> so there could occur races in `mkdir` and `sed`. The same is true
> when you try to build several images at the same time:
> 
>    $ bitbake image1 image2
>
> IIRC bitbake anonymous tasks are executed during parsing stage, so
> they are serialized.

I see, but that makes them a very strong measure that you can never
override i.e. in a layer on top. Not sure if that matters but i would
shy away from using the strongest measure if it can be done in a
different way.

For the dir how about using newtask[dirs] i guess bitbake will know how
to handle that and it seems to be the bitbake way anyways.

And for the sed, if the race happens and two or more seds get run. They
should just produce the same output and not fail if it was already there

sed "s#{DISTRO_NAME}#" + d.getVar("DEBDISTRONAME", True) + "#g" \
	cache_conf_file_in > cache_conf_file

Henning

> Alex
> 
> > Henning
> >   
> >>   do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}"
> >>   
> >>   # Install Debian packages, that were built from sources  
> >