Re: [PATCH 2/2] centralize multistrap configuration generation

[Alexander Smirnov <asmirnov@ilbers.de>]

On 02/16/2018 12:31 PM, Claudius Heine wrote:
> On 02/15/2018 02:39 PM, Alexander Smirnov wrote:
>>
>>
>> On 02/15/2018 04:20 PM, Claudius Heine wrote:
>>> On 02/15/2018 01:37 PM, Alexander Smirnov wrote:
>>>>
>>>> Claudius Heine <claudius.heine.ext@siemens.com> 15 февраля 2018 г.
>>>> 15:08:50 написал:
>>>>
>>>>> Hi Baurzhan,
>>>>>
>>>>> On 02/15/2018 12:42 PM, Baurzhan Ismagulov wrote:
>>>>>> On Wed, Feb 14, 2018 at 02:15:09PM +0100,
>>>>>> claudius.heine.ext@siemens.com wrote:
>>>>>>> From: Claudius Heine <ch@denx.de>
>>>>>>>
>>>>>>> Previous there was redundant code between the buildchroot and
>>>>>>> isar-image-base recipe.
>>>>>>>
>>>>>>> This patch moves the generation of the multistrap configuration to a
>>>>>>> central python script that can be parametrized by both recipes.
>>>>>>>
>>>>>>> Signed-off-by: Claudius Heine <ch@denx.de>
>>>>>>> ---
>>>>>>>    .../recipes-core/images/files/multistrap.conf.in   | 38 ----------
>>>>>>>    meta/lib/isar/multistrap.py                        | 80
>>>>>>> ++++++++++++++++++++++
>>>>>>>    .../buildchroot/files/multistrap.conf.in           | 37 ----------
>>>>>>
>>>>>> Could you please post two final multistrap.confs you'd like to get
>>>>>> with your
>>>>>> changes?
>>>>>
>>>>> 'like to get'? I don't want to get any other multistrap.conf than
>>>>> before. This is not a change of the multistrap.conf files, but just how
>>>>> they are generated. The python configparser created ini-file format
>>>>> only
>>>>> puts spaces around the '=' characters, but the resulting deserialized
>>>>> objects is the same.
>>>>
>>>> The difference that I see IMHO: now you need to feed multistrap by
>>>> community-standardized input, you propose to add custom tool that
>>>> eats proprietary input.
>>>
>>> I don't get this point. What do you mean by that?
>>
>> You've introduced new language, like:
>>
>> +MULTISTRAP_ISAR_SOURCE = "copy:///${DEPLOY_DIR_APT}/${DISTRO}"
>> +MULTISTRAP_ISAR_INSTALL = "${IMAGE_INSTALL}"
>> +MULTISTRAP_ISAR_SUITE = "${DEBDISTRONAME}"
>>
>> So to have working multistrap config I *have* to write pre-config in
>> this new language, what it's the simplification here, one set of options
>> is replaced by another...
> 
> ??? Thats not a 'new language' that just bitbake variables. I think I
> don't get what you are saying. Thats the same stuff that previous to
> this change happend in this big wall of text sed command. IMO that this
> is much more to the point and therefore easier to handle than this big
> search and replace list.

1. I suppose your parser eats only *specific* variables, so there is 
predefined list of extra variables I should somehow learn.

2. Each variable has some meaning, so depending on what I want from 
multistrap, I should *manually* assign them. For example if I want to 
use native arch, I have to set something.

3. These variables are complete duplication of multistrap config tokens.

So all the above looks like: instead of setting variables directly in 
multistrap config, you prefer to set them in recipe. A matter of taste, 
but for me this sounds like unnecessary complication of existing approach.

If everything here is only about avoiding duplication, so the multistrap 
cascading configuration [1] could be used. Common multistrap config part 
could be split in separate file and included by all rootfs generators.

[1] - respective section in multistrap man page.

> 
>>>
>>>>
>>>>>
>>>>> Ok, that is not 100% true, I changed the name of the isar repository
>>>>> section. Now its called 'isar' in the image (previously 'Isar') and
>>>>> buildchroot (previously 'isar-apt') multiconfig. But that is it. It has
>>>>> no different effect outside of multistrap.
>>>>>
>>>>>> I'd like to understand the reasons for moving from templates to a
>>>>>> magic
>>>>>> generator.
>>>>>>
>>>>>> In my experience, magic is usually poorly documented (if at all)
>>>>>> and very
>>>>>> ununderstandable for users. Simple change of a line requires
>>>>>> locating the
>>>>>> generator, understanding what it does (they tend to be hairy with
>>>>>> tons of
>>>>>> conditions) and patching the upstream [Isar] code. Templates have
>>>>>> their
>>>>>> disadvantages, but are straightforward for users. In general, I'd
>>>>>> like to keep
>>>>>> the current UX and stick with templates, which is possible most of
>>>>>> the time.
>>>>>
>>>>> You prefer duplicated code? Two slightly different templates for two
>>>>> buildchroot and image recipe?
>>>>>
>>>>
>>>> AFAIK the key idea is to have the only single multistrap that works
>>>> with upstream repos. Other multistraps fetch packages from local
>>>> cache. So after this is done, the problem with duplicacations in
>>>> configs will go.
>>>>
>>>>> Also this patch is only an intermediate step of the way for multi-repo
>>>>> support. IMO it has value on its own, since it centralized the
>>>>> generation of the multistrap configuration.
>>>>> The current kind of 'template' does not support creating repository
>>>>> entries on the fly.
>>>>
>>>> What is the use case to have them "on the fly"?
>>>
>>> See: Also this patch is only an intermediate step of the way for
>>> multi-repo support.
>>>
>>> Use-case for multi-repo: docker-ce repository available.
>>
>> Sorry, don't know what is docker-ce. Could you please describe whole
>> repositories layout, just to understand the topic?
> 
> I have the need to install docker community edition from the developers
> repository in my image. Currently I do it by downloading the 'deb' file
> from the repo manually and installing it, here I have to take care of
> updating the version to the latest one. What would be much better if I
> can just add this repository to the multistrap process and install the
> most current version like that. For that I need the ability to add
> additional repositories to the multistrap configuration from other layers.

Why you can't add respective section to multistrap.conf.in?

> 
> I don't understand what you mean by 'whole repositories layout'. You
> mean like a tree, but from what repository?

Let's say, final mulstisrap configuration file or the list of sources in 
/etc/apt.

Alex

> 
>>
>>>
>>>>
>>>> Alex
>>>>
>>>> It might be possible to extent the multiconfig with
>>>>> shell scripting, but I prefer using python for such advanced logic.
>>>>
>>>> I don't care about python or bash >
>>> At least Baurzhan seems to care, because he doesn't like debootstrap
>>> because of it being in shell.
>>
>> Aghr, sent the mail on interrupted line, sorry. :-(
>>
>> I meant I don't care whether python or bash is used, my concern what is
>> the input for they. If you prefer to parse multistrap config with python
>> instead of bash, I could understand this.
> 
> I don't understand you again. I don't parse the multistrap config. I
> just generate one. >
>> But here you want to introduce another language for doing the same things.
> 
> Again, what other language? I don't remember writing or using any
> language parse in this patch.
> 
>> So at least for me, it would be much easier to take some template for
>> multistrap config (which syntax is well known and available in man page)
>> and create the custom layout that I need.
> 
> If you prefer the template approach, then we might need to introduce a
> new templating language, because only search and replace via sed doesn't
> cut it for the multi-repo feature.
> 
> Since I know jinja [1] we could do it with this. But it still might
> result in something more hacky.
> 
> [1] http://jinja.pocoo.org/
> 
> Also since Jan, already decreed the switch to debootstrap I will now
> continue working on a patchset for this.

I still don't understand why it's so difficult to create new section in 
multistrap.conf file manually? Moreover that's not the place in Isar 
which is going to be changed tens times per-build each build. Once 
you've changed it, you likely never will need to this again within 
current project.

In general, it's impossible to get automatically configure upstream Isar 
according to your requirements. You some how need to specify what you 
want to have via some configuration interface: config files, variables, 
recipes etc...

What is the value to change one manual work (populating 
multistrap.conf.in) to another (populating MULTISTRAP_* variables)?

Alex

> 
> Cheers,
> Claudius
> 
>>
>> Alex
>>
>>>
>>> Cheers,
>>> Claudius
>>>
>>>>
>>>>>
>>>>> regards,
>>>>> Claudius
>>>>>
>>>>> -- 
>>>>> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
>>>>> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
>>>>> Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de
>>>>>
>>>>> -- 
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "isar-users" group.
>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>> send an email to isar-users+unsubscribe@googlegroups.com.
>>>>> To post to this group, send email to isar-users@googlegroups.com.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/isar-users/d58343b9-c284-81e7-f743-2672c1f5f8ef%40siemens.com.
>>>>>
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>>>
>>>
>>
> 

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov