Why does Isar build multiple configs in one OUTDIR?

Why does Isar build multiple configs in one OUTDIR?

[Henning Schild]

Hey,

at first i did not get the whole "extra-stamp" and ultra-deep directory
structure /DISTRO/ARCH/... But now i got it, Isar actually builds
multiple configs in a shared output directory. And in order to do so it
has to make sure that every single file does not create a name
collision. Coming up with extra-stamps and other funny stuff.

That way accidental reuse of any sort of build result is avoided. All
recipes inherit that complication and have to play along i.e. when
creating temporary files. Avoiding clashes is a very hard job to get
right throughout the whole system.

And at the end there seems to be absolutely no reuse of build results,
which would be the only justification for such madness. All Isar does
is come up with long names to avoid such reuse.

If you are building a kernel for two targets you use two O=, same is
true for cmake-projects and many other build systems out there.

The only reuse of a build step i see at the moment is the git-clone of
Isar itself. Where are the gains that justify the complexity? If there
is a good reason, that i did understand yet, please answer to this
point and you can stop reading here.

IMHO what should happen is:
- you clone your Isar once
- you clone your multiple target layers to _different_ dirs
- you run many (parallel) bitbakes in these dirs
- you _never_ build two targets in the same tree
- we drop all the complexity and do not force it down into
  end-users recipes

I think there is the one use-case where you build two very same images
with the same ARCH DISTRO etc. and actually want to reuse. That
probably already works out of the box and if it does not, coming up
with long stamps and directory names that carry the shared attributes
does not help at all.

Henning

Re: Why does Isar build multiple configs in one OUTDIR?

[Alexander Smirnov]

Henning Schild <henning.schild@siemens.com> 10 августа 2017 г. 10:09:06 
написал:

> Hey,
>
> at first i did not get the whole "extra-stamp" and ultra-deep directory
> structure /DISTRO/ARCH/... But now i got it, Isar actually builds
> multiple configs in a shared output directory. And in order to do so it
> has to make sure that every single file does not create a name
> collision. Coming up with extra-stamps and other funny stuff.

It's not a "funny" stuff, it's how almost all the build systems works, 
including Yocto and OE. Please avoid "labels" in your comments.

>
> That way accidental reuse of any sort of build result is avoided.

That's not true. Build for two machines with similar architecture and 
distro will reuse buildchroot and build results (if there are no machine 
specific stuff)

> recipes inherit that complication and have to play along i.e. when
> creating temporary files. Avoiding clashes is a very hard job to get
> right throughout the whole system.

Bitbake already provides *dedicated* way to do this, so we just use it.

>
> And at the end there seems to be absolutely no reuse of build results,

Again, that's not true. You probably speak about current implementation, 
where we have sample machines to demontrate PoC. But Isar in general is a 
tool to build your product and main design requirements have come from real 
projects.

> which would be the only justification for such madness. All Isar does
> is come up with long names to avoid such reuse.

That's the way how bitbake works. As I said, Yocto and OE uses the same 
approach.

>
> If you are building a kernel for two targets you use two O=, same is
> true for cmake-projects and many other build systems out there.
>
> The only reuse of a build step i see at the moment is the git-clone of
> Isar itself. Where are the gains that justify the complexity? If there
> is a good reason, that i did understand yet, please answer to this
> point and you can stop reading here.
>
> IMHO what should happen is:
> - you clone your Isar once
> - you clone your multiple target layers to _different_ dirs
> - you run many (parallel) bitbakes in these dirs
> - you _never_ build two targets in the same tree
> - we drop all the complexity and do not force it down into
>   end-users

I suppose that you are speaking about single use where you have specified 
machine and distro. One of the initial Isar requirement was to have 
possibility to support whole product, where product can have different 
platform and different base-system version. To release this product, you 
should build everything at once. Fetching various directories, setup them 
etc... bring unnecessary effort and complexity to whole product 
maintenance. Product is a single unit, that can't be split into the layers. 
And as bigger product machine/distro matrix, as much more complicated to 
support it.

Alex

>
> I think there is the one use-case where you build two very same images
> with the same ARCH DISTRO etc. and actually want to reuse. That
> probably already works out of the box and if it does not, coming up
> with long stamps and directory names that carry the shared attributes
> does not help at all.
>
> Henning

Re: Why does Isar build multiple configs in one OUTDIR?

[Henning Schild]

Am Thu, 10 Aug 2017 14:22:14 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Henning Schild <henning.schild@siemens.com> 10 августа 2017 г.
> 10:09:06 написал:
> 
> > Hey,
> >
> > at first i did not get the whole "extra-stamp" and ultra-deep
> > directory structure /DISTRO/ARCH/... But now i got it, Isar
> > actually builds multiple configs in a shared output directory. And
> > in order to do so it has to make sure that every single file does
> > not create a name collision. Coming up with extra-stamps and other
> > funny stuff.  
> 
> It's not a "funny" stuff, it's how almost all the build systems
> works, including Yocto and OE. Please avoid "labels" in your comments.
> 
> >
> > That way accidental reuse of any sort of build result is avoided.  
> 
> That's not true. Build for two machines with similar architecture and 
> distro will reuse buildchroot and build results (if there are no
> machine specific stuff)

Is that the point i covered in the last section?

> > recipes inherit that complication and have to play along i.e. when
> > creating temporary files. Avoiding clashes is a very hard job to get
> > right throughout the whole system.  
> 
> Bitbake already provides *dedicated* way to do this, so we just use
> it.
> 
> >
> > And at the end there seems to be absolutely no reuse of build
> > results,  
> 
> Again, that's not true. You probably speak about current
> implementation, where we have sample machines to demontrate PoC. But
> Isar in general is a tool to build your product and main design
> requirements have come from real projects.
> 
> > which would be the only justification for such madness. All Isar
> > does is come up with long names to avoid such reuse.  
> 
> That's the way how bitbake works. As I said, Yocto and OE uses the
> same approach.
> 
> >
> > If you are building a kernel for two targets you use two O=, same is
> > true for cmake-projects and many other build systems out there.
> >
> > The only reuse of a build step i see at the moment is the git-clone
> > of Isar itself. Where are the gains that justify the complexity? If
> > there is a good reason, that i did understand yet, please answer to
> > this point and you can stop reading here.

Please answer this key question, with examples.

> > IMHO what should happen is:
> > - you clone your Isar once
> > - you clone your multiple target layers to _different_ dirs
> > - you run many (parallel) bitbakes in these dirs
> > - you _never_ build two targets in the same tree
> > - we drop all the complexity and do not force it down into
> >   end-users  
> 
> I suppose that you are speaking about single use where you have
> specified machine and distro. One of the initial Isar requirement was
> to have possibility to support whole product, where product can have
> different platform and different base-system version.

What is a "product"? I would say the Iphone is a product and you would
have 7 configs for its respective versions. I can see how you would
want to build those 7 in one tree. What you are suggesting is the whole
"product line" of Apple in one "product", from the smalles ipod to the
workstations.

> To release this
> product, you should build everything at once. Fetching various
> directories, setup them etc... bring unnecessary effort and
> complexity to whole product maintenance.

Wrap the building with a script, i.e. have a look at kas.

> Product is a single unit,
> that can't be split into the layers. And as bigger product
> machine/distro matrix, as much more complicated to support it.

How can one atomic thing have multiple architechtures, i do not get
that point at all! Is your product a whole building or airplain? Even
if it was it needs to be modular ...

Henning

> Alex
> 
> >
> > I think there is the one use-case where you build two very same
> > images with the same ARCH DISTRO etc. and actually want to reuse.
> > That probably already works out of the box and if it does not,
> > coming up with long stamps and directory names that carry the
> > shared attributes does not help at all.
> >
> > Henning  
> 
> 

Re: Why does Isar build multiple configs in one OUTDIR?

[Alexander Smirnov]

On 08/10/2017 03:17 PM, Henning Schild wrote:
> Am Thu, 10 Aug 2017 14:22:14 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> Henning Schild <henning.schild@siemens.com> 10 августа 2017 г.
>> 10:09:06 написал:
>>
>>> Hey,
>>>
>>> at first i did not get the whole "extra-stamp" and ultra-deep
>>> directory structure /DISTRO/ARCH/... But now i got it, Isar
>>> actually builds multiple configs in a shared output directory. And
>>> in order to do so it has to make sure that every single file does
>>> not create a name collision. Coming up with extra-stamps and other
>>> funny stuff.
>>
>> It's not a "funny" stuff, it's how almost all the build systems
>> works, including Yocto and OE. Please avoid "labels" in your comments.
>>
>>>
>>> That way accidental reuse of any sort of build result is avoided.
>>
>> That's not true. Build for two machines with similar architecture and
>> distro will reuse buildchroot and build results (if there are no
>> machine specific stuff)
> 
> Is that the point i covered in the last section?
> 
>>> recipes inherit that complication and have to play along i.e. when
>>> creating temporary files. Avoiding clashes is a very hard job to get
>>> right throughout the whole system.
>>
>> Bitbake already provides *dedicated* way to do this, so we just use
>> it.
>>
>>>
>>> And at the end there seems to be absolutely no reuse of build
>>> results,
>>
>> Again, that's not true. You probably speak about current
>> implementation, where we have sample machines to demontrate PoC. But
>> Isar in general is a tool to build your product and main design
>> requirements have come from real projects.
>>
>>> which would be the only justification for such madness. All Isar
>>> does is come up with long names to avoid such reuse.
>>
>> That's the way how bitbake works. As I said, Yocto and OE uses the
>> same approach.
>>
>>>
>>> If you are building a kernel for two targets you use two O=, same is
>>> true for cmake-projects and many other build systems out there.
>>>
>>> The only reuse of a build step i see at the moment is the git-clone
>>> of Isar itself. Where are the gains that justify the complexity? If
>>> there is a good reason, that i did understand yet, please answer to
>>> this point and you can stop reading here.
> 
> Please answer this key question, with examples.
> 
>>> IMHO what should happen is:
>>> - you clone your Isar once
>>> - you clone your multiple target layers to _different_ dirs
>>> - you run many (parallel) bitbakes in these dirs
>>> - you _never_ build two targets in the same tree
>>> - we drop all the complexity and do not force it down into
>>>    end-users
>>
>> I suppose that you are speaking about single use where you have
>> specified machine and distro. One of the initial Isar requirement was
>> to have possibility to support whole product, where product can have
>> different platform and different base-system version.
> 
> What is a "product"? I would say the Iphone is a product and you would
> have 7 configs for its respective versions. I can see how you would
> want to build those 7 in one tree. What you are suggesting is the whole
> "product line" of Apple in one "product", from the smalles ipod to the
> workstations.

Product is what you sell in the market. From technical point of view, 
product can be described as a set of 3 parts: hardware, base system, 
application layer. In typical cases, you maintain the application layer 
only, the other things are provided by suppliers (internal or external). 
Releasing new version of your application layer forces you to provide 
software update for whole product-line officially supported.

> 
>> To release this
>> product, you should build everything at once. Fetching various
>> directories, setup them etc... bring unnecessary effort and
>> complexity to whole product maintenance.
> 
> Wrap the building with a script, i.e. have a look at kas.

Why each Isar user should wrap such scripts for each project, when this 
can be a part of basic functionality? Does this really make Isar less 
complicated?

I'd agree with your concern in context, that this design can be overhead 
for case where we have *single* machine and *single* distro version. But 
initial Isar idea was to support products with matrix of configurations.

> 
>> Product is a single unit,
>> that can't be split into the layers. And as bigger product
>> machine/distro matrix, as much more complicated to support it.
> 
> How can one atomic thing have multiple architechtures, i do not get
> that point at all! Is your product a whole building or airplain? Even
> if it was it needs to be modular ...
> 
> Henning
> 
>> Alex
>>
>>>
>>> I think there is the one use-case where you build two very same
>>> images with the same ARCH DISTRO etc. and actually want to reuse.
>>> That probably already works out of the box and if it does not,
>>> coming up with long stamps and directory names that carry the
>>> shared attributes does not help at all.
>>>
>>> Henning
>>
>>
> 

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

Re: Why does Isar build multiple configs in one OUTDIR?

[Baurzhan Ismagulov]

On Thu, Aug 10, 2017 at 02:17:54PM +0200, Henning Schild wrote:
> > > The only reuse of a build step i see at the moment is the git-clone
> > > of Isar itself. Where are the gains that justify the complexity? If
> > > there is a good reason, that i did understand yet, please answer to
> > > this point and you can stop reading here.
> 
> Please answer this key question, with examples.

Alex did.

Industry Control Hub Workshop Edition is armhf and uses jessie.
Industry Control Hub Plant Edition is armhf and uses wheezy.
They share the same user-space application.
The application is updated.
Both models are released.
You build them from the same build dir.

This comes from real life and this has been the key requirement on Isar from
the very beginning. The customer wants to build all artifacts with the same
release cycle in one step.

If we returned to the pre-multiconfig way of having separate build dirs,
automating building all products in the repo would require gluing those two
steps together. How? Shell with sed patching of local.conf? What for? Isar *is*
the single-stop glue.

With kind regards,
Baurzhan.

Re: Why does Isar build multiple configs in one OUTDIR?

[Henning Schild]

Am Thu, 10 Aug 2017 15:09:51 +0200
schrieb Baurzhan Ismagulov <ibr@radix50.net>:

> On Thu, Aug 10, 2017 at 02:17:54PM +0200, Henning Schild wrote:
> > > > The only reuse of a build step i see at the moment is the
> > > > git-clone of Isar itself. Where are the gains that justify the
> > > > complexity? If there is a good reason, that i did understand
> > > > yet, please answer to this point and you can stop reading
> > > > here.  
> > 
> > Please answer this key question, with examples.  
> 
> Alex did.

Well I did not understand it. And i still do not.

> Industry Control Hub Workshop Edition is armhf and uses jessie.
> Industry Control Hub Plant Edition is armhf and uses wheezy.
> They share the same user-space application.

So two complete rootfs s that share one application? And they probably
only share the source and recipe to build it.
I still do not see where these two share any (intermediate) build
results, sorry.

> The application is updated.
> Both models are released.
> You build them from the same build dir.

?

> This comes from real life and this has been the key requirement on
> Isar from the very beginning. The customer wants to build all
> artifacts with the same release cycle in one step.
> 
> If we returned to the pre-multiconfig way of having separate build
> dirs, automating building all products in the repo would require
> gluing those two steps together. How? Shell with sed patching of
> local.conf? 

Bitbake Layering. Maybe combined with git submodules or kas.

> What for?

Simplicity, Elegance, Maintainability, Modularity ... ty

> Isar *is* the single-stop glue.

Ok. Understood but not convinced.

Henning

> With kind regards,
> Baurzhan.
> 

Re: Why does Isar build multiple configs in one OUTDIR?

[Baurzhan Ismagulov]

On Thu, Aug 10, 2017 at 03:52:37PM +0200, Henning Schild wrote:
> > Industry Control Hub Workshop Edition is armhf and uses jessie.
> > Industry Control Hub Plant Edition is armhf and uses wheezy.
> > They share the same user-space application.
> 
> So two complete rootfs s that share one application? And they probably
> only share the source and recipe to build it.
> I still do not see where these two share any (intermediate) build
> results, sorry.

ICH Plant Edition contains multiple cards that are armhf, use wheezy but have
different user-space applications. They share the same buildchroot.


> > If we returned to the pre-multiconfig way of having separate build
> > dirs, automating building all products in the repo would require
> > gluing those two steps together. How? Shell with sed patching of
> > local.conf? 
> 
> Bitbake Layering. Maybe combined with git submodules or kas.

Maybe? Please specify how you suggest to build images for ICH WE and PE.

The fact is, the products are already layered and split in different git repos.
Apart from the fact that Isar predates kas and solves the problem well: I'd use
kas for cloning disparate repos (like the repo tool in Android), not for build
dir management.


> > What for?
> 
> Simplicity, Elegance, Maintainability, Modularity ... ty

After you specify your image building solution, please show how yours is better
in these regards.

If you need only one arch + suite combination, you may still use local.conf
without multiconfig.

And if you like, you may still layout your product to your taste and use your
preferred solution with stock Isar. What is exactly the problem you are trying
to address? Multiconfig doesn't take anything away from you. Artifact reuse is
in place within the same arch + suite.


With kind regards,
Baurzhan.

Re: Why does Isar build multiple configs in one OUTDIR?

[Jan Kiszka]

On 2017-08-10 10:25, Baurzhan Ismagulov wrote:
>>> If we returned to the pre-multiconfig way of having separate build
>>> dirs, automating building all products in the repo would require
>>> gluing those two steps together. How? Shell with sed patching of
>>> local.conf? 
>>
>> Bitbake Layering. Maybe combined with git submodules or kas.
> 
> Maybe? Please specify how you suggest to build images for ICH WE and PE.
> 
> The fact is, the products are already layered and split in different git repos.
> Apart from the fact that Isar predates kas and solves the problem well: I'd use
> kas for cloning disparate repos (like the repo tool in Android), not for build
> dir management.

kas manages config dependencies that are under the same release regime
in a single file. It also provides a clean, archiveable build
environment, which is even more important with OE.

If your production generates multiple variants from a single release,
there is surely a value in exploiting the new multiconfig feature of
bitbake. It's an optimization that can save a hand full of lines in your
CI script and likely quite a bit cycles of your CI server. I see it as
an added value. But as Isar is already wrapped around it, no longer
works without it (because that would mean duplicate maintenance), we
have to make sure to support it, including a better support in kas.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: Why does Isar build multiple configs in one OUTDIR?

[Henning Schild]

Am Thu, 10 Aug 2017 16:25:02 +0200
schrieb Baurzhan Ismagulov <ibr@radix50.net>:

> On Thu, Aug 10, 2017 at 03:52:37PM +0200, Henning Schild wrote:
> > > Industry Control Hub Workshop Edition is armhf and uses jessie.
> > > Industry Control Hub Plant Edition is armhf and uses wheezy.
> > > They share the same user-space application.  
> > 
> > So two complete rootfs s that share one application? And they
> > probably only share the source and recipe to build it.
> > I still do not see where these two share any (intermediate) build
> > results, sorry.  
> 
> ICH Plant Edition contains multiple cards that are armhf, use wheezy
> but have different user-space applications. They share the same
> buildchroot.

Ok that is the example i asked about initially. Thanks!

Henning

> 
> > > If we returned to the pre-multiconfig way of having separate build
> > > dirs, automating building all products in the repo would require
> > > gluing those two steps together. How? Shell with sed patching of
> > > local.conf?   
> > 
> > Bitbake Layering. Maybe combined with git submodules or kas.  
> 
> Maybe? Please specify how you suggest to build images for ICH WE and
> PE.
> 
> The fact is, the products are already layered and split in different
> git repos. Apart from the fact that Isar predates kas and solves the
> problem well: I'd use kas for cloning disparate repos (like the repo
> tool in Android), not for build dir management.
> 
> 
> > > What for?  
> > 
> > Simplicity, Elegance, Maintainability, Modularity ... ty  
> 
> After you specify your image building solution, please show how yours
> is better in these regards.
> 
> If you need only one arch + suite combination, you may still use
> local.conf without multiconfig.
> 
> And if you like, you may still layout your product to your taste and
> use your preferred solution with stock Isar. What is exactly the
> problem you are trying to address? Multiconfig doesn't take anything
> away from you. Artifact reuse is in place within the same arch +
> suite.
> 
> 
> With kind regards,
> Baurzhan.
> 

Re: Why does Isar build multiple configs in one OUTDIR?

[Baurzhan Ismagulov]

On Thu, Aug 10, 2017 at 12:37:01PM -0400, Jan Kiszka wrote:
> kas manages config dependencies that are under the same release regime
> in a single file. It also provides a clean, archiveable build
> environment, which is even more important with OE.
> 
> If your production generates multiple variants from a single release,
> there is surely a value in exploiting the new multiconfig feature of
> bitbake. It's an optimization that can save a hand full of lines in your
> CI script and likely quite a bit cycles of your CI server. I see it as
> an added value. But as Isar is already wrapped around it, no longer
> works without it (because that would mean duplicate maintenance), we
> have to make sure to support it, including a better support in kas.

Just to ensure we are talking about the same thing: BitBake's multiconfig
feature is a means of building for multiple MACHINE + DISTRO configurations in
a single run without having to deal with local.conf in between. Multiconfig and
local.conf are completely equivalent and mutually interchangeable. BitBake and
Isar support either without requiring the other, or both. In the same build
dir, one can use local.conf for one bitbake run and multiconfig for another;
the results are the same. We refer to multiconfig in the user manual to avoid
lengthier textual description of editing local.conf.

I agree with both of your points:

* Combining two non-multiconfig builds via external tools would add maintenance
  overhead, and

* kas's goal is CM of disparate entities under the same release regime rather
  than dealing with build issues.

With kind regards,
Baurzhan.

Re: Why does Isar build multiple configs in one OUTDIR?

[Jan Kiszka]

On 2017-08-11 14:20, Baurzhan Ismagulov wrote:
> On Thu, Aug 10, 2017 at 12:37:01PM -0400, Jan Kiszka wrote:
>> kas manages config dependencies that are under the same release regime
>> in a single file. It also provides a clean, archiveable build
>> environment, which is even more important with OE.
>>
>> If your production generates multiple variants from a single release,
>> there is surely a value in exploiting the new multiconfig feature of
>> bitbake. It's an optimization that can save a hand full of lines in your
>> CI script and likely quite a bit cycles of your CI server. I see it as
>> an added value. But as Isar is already wrapped around it, no longer
>> works without it (because that would mean duplicate maintenance), we
>> have to make sure to support it, including a better support in kas.
> 
> Just to ensure we are talking about the same thing: BitBake's multiconfig
> feature is a means of building for multiple MACHINE + DISTRO configurations in
> a single run without having to deal with local.conf in between. Multiconfig and
> local.conf are completely equivalent and mutually interchangeable. BitBake and
> Isar support either without requiring the other, or both. In the same build

Theoretically, Isar could still build without multiconfig. But
practically, more and more information is now only in
meta-isar/conf/multiconfig, and one would has to copy that out into an
own local.conf (or a kas config).

But focusing on one method is fine, we should just avoid confusions
regarding the other.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: Why does Isar build multiple configs in one OUTDIR?

[Baurzhan Ismagulov]

On Sat, Aug 19, 2017 at 09:53:54AM -0400, Jan Kiszka wrote:
> Theoretically, Isar could still build without multiconfig. But
> practically, more and more information is now only in
> meta-isar/conf/multiconfig, and one would has to copy that out into an
> own local.conf (or a kas config).

Good catch, thanks. Indeed, building with the default local.conf produces a
working image, albeit the kernel / initrd is not extracted.
https://github.com/ilbers/isar/issues/29


> But focusing on one method is fine, we should just avoid confusions
> regarding the other.

As multiconfig is relatively new in bitbake, many people are not familiar with
it. Besides, local.conf would save people with one config the cumbersome
multiconfig syntax.

I think keeping local.conf working shouldn't cost too much effort -- in the
worst case, the vars added to multiconfig should also be added to local.conf.
Maybe we manage to come up with some standard customization file location
without having to repeat ourselves.


With kind regards,
Baurzhan.