[RFC] can we just extend openembedded to get Isar 2.0

[RFC] can we just extend openembedded to get Isar 2.0

[Henning Schild]

Hi,

looking at Isar it basically is a bitbake recipe to extract a bunch
of .debs into a rootfs. It also has support for a few more things, like
(cross-)compiling your own packages, basic configuration etc. Things
like image creation with wic have been added as well.

But a lot of things are still missing. For true customization one would
need a way to smuggle files into the rootfs, patch files in there,
execute scripts in there. Ideally with collision protection and special
protection for config-files etc. Claudius already suggested building
custom .debs in Isar to solve that.

Another thing i already came across is Image conversion .raw
-> .qcow2/.vdi/.vmdk all that is missing in Isar. Same was true for the
wic support until recently.

People that are coming from Yocto and want to switch to Isar will never
get their recipes to work, because all the utils-classes are missing in
Isar and would need to be ported/pulled-in.

Isar needs sudo, and someone has to integrate libpseudo.

Isar still has several small issues with correct rebuilds after config
changes etc.

OE solved all those problems already, but what it can not do is
fetch .debs from a mirror and bundle them together with multistrap. OE
most likely contains a lot of other usefull stuff that we might need
eventually. But without a doubt it also contains tons of stuff that we
do not need and that might confuse users if the have to look into it.

Does it really make sense for us to reinvent all those OE-features in
Isar, or should we just got the other way around and put the Isar
features into OE?

At a first glance it looks like you have to teach OE to get debian
packages and create a rootfs from them. Tell it to not compile anything
at first. Probably much like the do_rootfs from Isar. We might get away
with a layer on top of OE and maybe a few patches to OE that can maybe
be mainlined. Or maybe the whole thing could become mainline OE some
day.

Has anyone considered or even tried that already? Going down that road
sounds like solving a lot of the open points in Isar at once, by
adding a few 100 LoC to OE. But i might be totally wrong here.
I guess everyone working on Isar needs a good technical answer to
why we seemingly start from scratch.

Henning

Re: [RFC] can we just extend openembedded to get Isar 2.0

[Claudius Heine]

[-- Attachment #1: Type: text/plain, Size: 3450 bytes --]

On Thu, 2017-07-20 at 10:30 +0200, Henning Schild wrote:
> Hi,
> 
> looking at Isar it basically is a bitbake recipe to extract a bunch
> of .debs into a rootfs. It also has support for a few more things,
> like
> (cross-)compiling your own packages, basic configuration etc. Things
> like image creation with wic have been added as well.
> 
> But a lot of things are still missing. For true customization one
> would
> need a way to smuggle files into the rootfs, patch files in there,
> execute scripts in there. Ideally with collision protection and
> special
> protection for config-files etc. Claudius already suggested building
> custom .debs in Isar to solve that.
> 
> Another thing i already came across is Image conversion .raw
> -> .qcow2/.vdi/.vmdk all that is missing in Isar. Same was true for
> the
> wic support until recently.
> 
> People that are coming from Yocto and want to switch to Isar will
> never
> get their recipes to work, because all the utils-classes are missing
> in
> Isar and would need to be ported/pulled-in.
> 
> Isar needs sudo, and someone has to integrate libpseudo.
> 
> Isar still has several small issues with correct rebuilds after
> config
> changes etc.
> 
> OE solved all those problems already, but what it can not do is
> fetch .debs from a mirror and bundle them together with multistrap.
> OE
> most likely contains a lot of other usefull stuff that we might need
> eventually. But without a doubt it also contains tons of stuff that
> we
> do not need and that might confuse users if the have to look into it.
> 
> Does it really make sense for us to reinvent all those OE-features in
> Isar, or should we just got the other way around and put the Isar
> features into OE?
> 
> At a first glance it looks like you have to teach OE to get debian
> packages and create a rootfs from them. Tell it to not compile
> anything
> at first. Probably much like the do_rootfs from Isar. We might get
> away
> with a layer on top of OE and maybe a few patches to OE that can
> maybe
> be mainlined. Or maybe the whole thing could become mainline OE some
> day.
> 
> Has anyone considered or even tried that already? Going down that
> road
> sounds like solving a lot of the open points in Isar at once, by
> adding a few 100 LoC to OE. But i might be totally wrong here.
> I guess everyone working on Isar needs a good technical answer to
> why we seemingly start from scratch.

I agree. Currently looking at the debian package creation code in
openembedded and thinking about how to reimplement this rather complex
and featureful code into isar is what triggered this line of thought
for me.

There is a lot of stuff in bitbake that is written for openembedded in
mind, so retrofitting it to a completely different build process seems
like an anti-pattern to me. Going the other way and integrating the
debian repositories as a binary package source into openembedded seems
more reasonable, and might be doable with the same or less amount of
code.

Cheers,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

            PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                              Keyserver: hkp://pool.sks-keyservers.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC] can we just extend openembedded to get Isar 2.0

[Jan Kiszka]

On 2017-07-20 10:30, Henning Schild wrote:
> Hi,
> 
> looking at Isar it basically is a bitbake recipe to extract a bunch
> of .debs into a rootfs. It also has support for a few more things, like
> (cross-)compiling your own packages, basic configuration etc. Things
> like image creation with wic have been added as well.
> 
> But a lot of things are still missing. For true customization one would
> need a way to smuggle files into the rootfs, patch files in there,
> execute scripts in there. Ideally with collision protection and special
> protection for config-files etc. Claudius already suggested building
> custom .debs in Isar to solve that.
> 
> Another thing i already came across is Image conversion .raw
> -> .qcow2/.vdi/.vmdk all that is missing in Isar. Same was true for the
> wic support until recently.
> 
> People that are coming from Yocto and want to switch to Isar will never
> get their recipes to work, because all the utils-classes are missing in
> Isar and would need to be ported/pulled-in.
> 
> Isar needs sudo, and someone has to integrate libpseudo.
> 
> Isar still has several small issues with correct rebuilds after config
> changes etc.
> 
> OE solved all those problems already, but what it can not do is
> fetch .debs from a mirror and bundle them together with multistrap. OE
> most likely contains a lot of other usefull stuff that we might need
> eventually. But without a doubt it also contains tons of stuff that we
> do not need and that might confuse users if the have to look into it.
> 
> Does it really make sense for us to reinvent all those OE-features in
> Isar, or should we just got the other way around and put the Isar
> features into OE?
> 
> At a first glance it looks like you have to teach OE to get debian
> packages and create a rootfs from them. Tell it to not compile anything
> at first. Probably much like the do_rootfs from Isar. We might get away
> with a layer on top of OE and maybe a few patches to OE that can maybe
> be mainlined. Or maybe the whole thing could become mainline OE some
> day.
> 
> Has anyone considered or even tried that already? Going down that road
> sounds like solving a lot of the open points in Isar at once, by
> adding a few 100 LoC to OE. But i might be totally wrong here.
> I guess everyone working on Isar needs a good technical answer to
> why we seemingly start from scratch.
> 
> Henning
> 

This is a very important architectural question as we need to decide now
where to focus our resources on: porting/implementing missing features
or enabling OE.

Baurzhan, Alex, I suspect you have thought about this before and may
provide some insights on the traps and pitfalls of an OE-core-based path.

I also take the freedom to add Yoshi and Daniel to this discussion to
bring in their meta-debian experience. Possibly, you solved some of the
issues regarding Debian with OE already, although you focus on
out-of-source building. Maybe this path would also allows us to move
even closer together. But that's still a 10000-meters view on the topic.

Thanks,
Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [RFC] can we just extend openembedded to get Isar 2.0

[Baurzhan Ismagulov]

Hello,

thanks for all the ideas.

The short answer is, Isar has started from scratch as a rootfs generator from
existing binary distro + self-built packages. In plain German ;) , OE is not
necessary for this purpose. OE is a distro, and Isar is a builder.

Of course, this is not that simple. You, and other people at various
conferences, ask about the tools that you know and want to use, and this is
exactly what we care about: Re-use mainstream tools and don't invent our own
wheel.

What we are doing now is exactly bringing the tools many people are familiar
with into Debian(-based system) development. Where the modified tools are
hosted (OE vs. Isar repos) is secondary; the primary issue is to make them
useful for both worlds -- one has to do that also if one inherits from OE.
Developing this same feature requires just the same effort compared to adapting
the OE tools for Debian directly in OE repos. Our plan is to harmonize the
tools for both use cases, submit the patches, and see what the OE community
thinks about that.

So, to answer your original question, I don't exclude the possibility of using
Isar on top of OE if it would ever make sense. However, keeping the possibility
of using plain Isar is important for commercial product builders. Rationale:
The short answer above, plus a major disadvantage: OE is huge, and this luggage
is a problem for CM and integrators (I had to maintain OE in ClearCase, and I'm
glad I don't anymore).

Regarding bitbake, I don't agree it's retrofitting. Bitbake is a great generic
tool, perfectly suitable for the task. In fact, we build other commercial
embedded systems with bitbake, without Isar or OE. Other tools like wic,
runqemu, etc. are more OE-centric (mostly due to sysroot), but I'm glad to see
that the required changes were quite manageable.

The problem with Debian support in OE is that it is very undebian. Debian has
its strengths, and there are reasons why Debian developers strongly dislike OE
(see e.g. [1], and I've heard many such discussions). Deby / meta-debian
follows the OE way of building Debian packages, and currently considers moving
to the Debian way. The reason is, as soon as you oeize a package, you have to
maintain it. That is why Isar does things the Debian way and aims at avoiding
massive package modifications; OE doesn't really help here.

A couple of comments below.

On Thu, Jul 20, 2017 at 11:15:09AM +0200, Jan Kiszka wrote:
> > But a lot of things are still missing. For true customization one would
> > need a way to smuggle files into the rootfs, patch files in there,
> > execute scripts in there. Ideally with collision protection and special
> > protection for config-files etc.

Customization is implemented in configscript.sh. Some more structure could be
introduced, but we didn't need it till now, as the changes beyond custom debs
were reasonable in size.


> > Claudius already suggested building custom .debs in Isar to solve that.

Yes. we built them where the effort was justified.


> > Another thing i already came across is Image conversion .raw
> > -> .qcow2/.vdi/.vmdk all that is missing in Isar. Same was true for the
> > wic support until recently.

As introduced above, Isar follows the Debian way to reuse as much as possible.
That is why just basing on OE won't make the tools work for the Isar layer OOTB
-- it will require the very adaptations we do now.


> > People that are coming from Yocto and want to switch to Isar will never
> > get their recipes to work, because all the utils-classes are missing in
> > Isar and would need to be ported/pulled-in.

This is what we ideally don't want to have. It is possible to build packages in
a basic OE way, but the preferred way is to debianize all packages. We want to
have stuff like clean, etc., but duplicating the whole OE functionality doesn't
make sense and won't work, since Isar and OE have different base systems and
toolchains.


> > Isar needs sudo, and someone has to integrate libpseudo.

This is on TODO.


> > Isar still has several small issues with correct rebuilds after config
> > changes etc.

These issues are inherited from bitbake and its default setting of doing so.
IIRC, there was a var to change that behavior, but I couldn't remember its
name, have to dig the code.


> > OE solved all those problems already

AFAIK, not the rebuild one.


> > but what it can not do is
> > fetch .debs from a mirror and bundle them together with multistrap. OE
> > most likely contains a lot of other usefull stuff that we might need
> > eventually.

You can copy the recipe from Isar, and OE will be able to multistrap. The
problem is, you can't do anything useful with that afterwards due to the
different base system and compiler. OE != Debian, just like Ubuntu != Debian.


> > Does it really make sense for us to reinvent all those OE-features in
> > Isar, or should we just got the other way around and put the Isar
> > features into OE?

The truth is, both ways are the same due to the reasons above (except where the
modified tools are hosted, which doesn't matter), and we are already going that
way. Doing on top of OE doesn't make the tools work OOTB. But it brings the OE
distro luggage, which is not necessary for Isar use cases.


> > At a first glance it looks like you have to teach OE to get debian
> > packages and create a rootfs from them. Tell it to not compile anything
> > at first. Probably much like the do_rootfs from Isar. We might get away
> > with a layer on top of OE and maybe a few patches to OE that can maybe
> > be mainlined. Or maybe the whole thing could become mainline OE some
> > day.

IIUC, not compiling while keeping the whole vars intact would make that a
mutant behaving against the expectations of the users. Changing the vars would
be creating a small Isar inside OE; I don't think this can be done in a layer
alone. Regarding mainlining -- we want to start with the tools. I doubt the two
paradigms (distro vs. builder) can / should be aligned into a single system.


1. https://lists.debian.org/debian-embedded/2016/04/msg00013.html


With kind regards,
Baurzhan.

Re: [RFC] can we just extend openembedded to get Isar 2.0

[Jan Kiszka]

On 2017-07-21 01:05, Baurzhan Ismagulov wrote:
> Hello,
> 
> thanks for all the ideas.
> 
> The short answer is, Isar has started from scratch as a rootfs generator from
> existing binary distro + self-built packages. In plain German ;) , OE is not
> necessary for this purpose. OE is a distro, and Isar is a builder.

OE-core is also a distro, but that is indeed the uninteresting part. We
are interested in all the stuff in meta/classes, lib, you-name-it. All
what is still missing and needs to be copied, adjusted, and MAINTAINED
separately in isar so far. This gap is still to large, and we need to
define a strategy now how to close it sustainably.

> 
> Of course, this is not that simple. You, and other people at various
> conferences, ask about the tools that you know and want to use, and this is
> exactly what we care about: Re-use mainstream tools and don't invent our own
> wheel.

It's not just tools, it's also their infrastructure. The tools are easy
to copy, but the infrastructure has dependencies that need to be
continuously adjusted.

The question is now, can we bend the OE-core towards building a Debian
distro in the Debian way with its pre-existing infrastructure, or do we
really have to fork it?

> 
> What we are doing now is exactly bringing the tools many people are familiar
> with into Debian(-based system) development. Where the modified tools are
> hosted (OE vs. Isar repos) is secondary; the primary issue is to make them
> useful for both worlds -- one has to do that also if one inherits from OE.
> Developing this same feature requires just the same effort compared to adapting
> the OE tools for Debian directly in OE repos. Our plan is to harmonize the
> tools for both use cases, submit the patches, and see what the OE community
> thinks about that.
> 
> So, to answer your original question, I don't exclude the possibility of using
> Isar on top of OE if it would ever make sense. However, keeping the possibility
> of using plain Isar is important for commercial product builders. Rationale:
> The short answer above, plus a major disadvantage: OE is huge, and this luggage
> is a problem for CM and integrators (I had to maintain OE in ClearCase, and I'm
> glad I don't anymore).

OE-core is also just 58K LOC - smaller than bitbake itself, thus Isar.
(And ClearCase is dead, just not buried everywhere ;).)

> 
> Regarding bitbake, I don't agree it's retrofitting. Bitbake is a great generic
> tool, perfectly suitable for the task. In fact, we build other commercial
> embedded systems with bitbake, without Isar or OE. Other tools like wic,
> runqemu, etc. are more OE-centric (mostly due to sysroot), but I'm glad to see
> that the required changes were quite manageable.
> 
> The problem with Debian support in OE is that it is very undebian. Debian has
> its strengths, and there are reasons why Debian developers strongly dislike OE
> (see e.g. [1], and I've heard many such discussions). Deby / meta-debian
> follows the OE way of building Debian packages, and currently considers moving
> to the Debian way. The reason is, as soon as you oeize a package, you have to
> maintain it. That is why Isar does things the Debian way and aims at avoiding
> massive package modifications; OE doesn't really help here.

The idea is not to move to the OE way of doing Debian, for sure. It's
about adding the right way to the OE-core tools and libs that are useful
or even mandatory for Isar instead of forking or reimplementing them now.

> 
> A couple of comments below.
> 
> On Thu, Jul 20, 2017 at 11:15:09AM +0200, Jan Kiszka wrote:
>>> But a lot of things are still missing. For true customization one would
>>> need a way to smuggle files into the rootfs, patch files in there,
>>> execute scripts in there. Ideally with collision protection and special
>>> protection for config-files etc.
> 
> Customization is implemented in configscript.sh. Some more structure could be
> introduced, but we didn't need it till now, as the changes beyond custom debs
> were reasonable in size.

The problem is that Isar wasn't applied on a larger scale yet. And it
was only tested against the bitbake-unlike template pattern, instead of
using layering consequently. We are doing both now, and the problems
surface. All fixable, but the question is, where to do that best.

> 
> 
>>> Claudius already suggested building custom .debs in Isar to solve that.
> 
> Yes. we built them where the effort was justified.

No, this is about making this feature available as library (class) so
that you write three lines of recipe and have a customization deb
package. That effort would be minimal for the user, and we only need to
solve this once properly. If OE-core cannot provide that, we will have
to implement it ourselves.

> 
> 
>>> Another thing i already came across is Image conversion .raw
>>> -> .qcow2/.vdi/.vmdk all that is missing in Isar. Same was true for the
>>> wic support until recently.
> 
> As introduced above, Isar follows the Debian way to reuse as much as possible.
> That is why just basing on OE won't make the tools work for the Isar layer OOTB
> -- it will require the very adaptations we do now.
> 

Can you provide a concrete code example? What all prevents using
upstream wic (including its image class) more as-is and forces us to
fork it?

> 
>>> People that are coming from Yocto and want to switch to Isar will never
>>> get their recipes to work, because all the utils-classes are missing in
>>> Isar and would need to be ported/pulled-in.
> 
> This is what we ideally don't want to have. It is possible to build packages in
> a basic OE way, but the preferred way is to debianize all packages. We want to
> have stuff like clean, etc., but duplicating the whole OE functionality doesn't
> make sense and won't work, since Isar and OE have different base systems and
> toolchains.

Again, concrete examples would help to underline this high-level
statement. We will be asked these questions over and over again, so we
need the arguments ready at hand.

> 
> 
>>> Isar needs sudo, and someone has to integrate libpseudo.
> 
> This is on TODO.
> 

It's actually WIP by us now - it's urgent because we need CI support,
and that's in conflict with the sudo workarounds.

> 
>>> Isar still has several small issues with correct rebuilds after config
>>> changes etc.
> 
> These issues are inherited from bitbake and its default setting of doing so.
> IIRC, there was a var to change that behavior, but I couldn't remember its
> name, have to dig the code.
> 
> 
>>> OE solved all those problems already
> 
> AFAIK, not the rebuild one.
>

But, e.g., it has proper support for SRC_URI and fetching, proxies,
mirrors, image types, and possibly more we need. Are all of these
feature so tangled up with OE assumptions that stubbing and adjusting
their dependencies would be worse than reimplementing them?

> 
>>> but what it can not do is
>>> fetch .debs from a mirror and bundle them together with multistrap. OE
>>> most likely contains a lot of other usefull stuff that we might need
>>> eventually.
> 
> You can copy the recipe from Isar, and OE will be able to multistrap. The
> problem is, you can't do anything useful with that afterwards due to the
> different base system and compiler. OE != Debian, just like Ubuntu != Debian.
> 

OK, that is a more concrete example to discuss: assumption of OE-core
about the compiler and the build environment (when we are building, the
exceptional case). Where do they bite us when reusing OE-core to
generate, e.g., images? Or fetching sources? Or any other case?

Yes, 90% of OE-core may be unused with Isar. But even 10% would still be
an order of magnitude more lines of code than Isar has right now. That's
the concern. If we have to port all that, or even just half of it over,
we need strong reasons, ideally documented.

> 
>>> Does it really make sense for us to reinvent all those OE-features in
>>> Isar, or should we just got the other way around and put the Isar
>>> features into OE?
> 
> The truth is, both ways are the same due to the reasons above (except where the
> modified tools are hosted, which doesn't matter), and we are already going that
> way. Doing on top of OE doesn't make the tools work OOTB. But it brings the OE
> distro luggage, which is not necessary for Isar use cases.
> 
> 
>>> At a first glance it looks like you have to teach OE to get debian
>>> packages and create a rootfs from them. Tell it to not compile anything
>>> at first. Probably much like the do_rootfs from Isar. We might get away
>>> with a layer on top of OE and maybe a few patches to OE that can maybe
>>> be mainlined. Or maybe the whole thing could become mainline OE some
>>> day.
> 
> IIUC, not compiling while keeping the whole vars intact would make that a
> mutant behaving against the expectations of the users. Changing the vars would
> be creating a small Isar inside OE; I don't think this can be done in a layer
> alone. Regarding mainlining -- we want to start with the tools. I doubt the two
> paradigms (distro vs. builder) can / should be aligned into a single system.
> 
> 
> 1. https://lists.debian.org/debian-embedded/2016/04/msg00013.html
> 
> 
> With kind regards,
> Baurzhan.
> 

Thanks,
Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [RFC] can we just extend openembedded to get Isar 2.0

[Baurzhan Ismagulov]

On Fri, Jul 21, 2017 at 09:31:05AM +0200, Jan Kiszka wrote:
> > The short answer is, Isar has started from scratch as a rootfs generator from
> > existing binary distro + self-built packages. In plain German ;) , OE is not
> > necessary for this purpose. OE is a distro, and Isar is a builder.
> 
> OE-core is also a distro, but that is indeed the uninteresting part.

Not just uninteresting, it's 16 MiB of metadata that needs to be CM'ed and
legally cleared.


> We are interested in all the stuff in meta/classes, lib, you-name-it. All
> what is still missing and needs to be copied, adjusted, and MAINTAINED
> separately in isar so far.

Copying is not an issue. Adjusting is required in both ways. For maintenance,
we already want to submit patches to Yocto / OE.

If one goes the OE layer way, the tools have to be adjusted anyway. But they
won't be adjusted all at once. So, some tools would work, others would be
present but won't work, creating a mess.


> This gap is still to large, and we need to define a strategy now how to close
> it sustainably.

The value of Isar is to provide convenient build environment for Debian-based
systems. There are many things we can do on the Debian support side that no
other tools do, and that should be the priority. Closing the gap with OE is
going to be the same effort in either way.


> > Of course, this is not that simple. You, and other people at various
> > conferences, ask about the tools that you know and want to use, and this is
> > exactly what we care about: Re-use mainstream tools and don't invent our own
> > wheel.
...
> The question is now, can we bend the OE-core towards building a Debian
> distro in the Debian way with its pre-existing infrastructure, or do we
> really have to fork it?

We don't fork it to have our own copies. We adjust them to provide the patches
back upstream. So, for those tools, Isar is just a temporary storage place.


> > So, to answer your original question, I don't exclude the possibility of using
> > Isar on top of OE if it would ever make sense. However, keeping the possibility
> > of using plain Isar is important for commercial product builders. Rationale:
> > The short answer above, plus a major disadvantage: OE is huge, and this luggage
> > is a problem for CM and integrators (I had to maintain OE in ClearCase, and I'm
> > glad I don't anymore).
> 
> OE-core is also just 58K LOC - smaller than bitbake itself, thus Isar.

I assume your LOC counting tool doesn't count 14 MiB of metadata?

Bitbake is technically in the Isar repository just to ship the right version to
any host version, but it isn't a part of Isar.


> (And ClearCase is dead, just not buried everywhere ;).)

:) That was just an example that one has to CM the stuff.


> > The problem with Debian support in OE is that it is very undebian. Debian has
> > its strengths, and there are reasons why Debian developers strongly dislike OE
> > (see e.g. [1], and I've heard many such discussions). Deby / meta-debian
> > follows the OE way of building Debian packages, and currently considers moving
> > to the Debian way. The reason is, as soon as you oeize a package, you have to
> > maintain it. That is why Isar does things the Debian way and aims at avoiding
> > massive package modifications; OE doesn't really help here.
> 
> The idea is not to move to the OE way of doing Debian, for sure. It's
> about adding the right way to the OE-core tools and libs that are useful
> or even mandatory for Isar instead of forking or reimplementing them now.

The idea is clear, the answer is above: The adjustment effort is the same. We
want to merge upstream, we don't reimplement.


> > Customization is implemented in configscript.sh. Some more structure could be
> > introduced, but we didn't need it till now, as the changes beyond custom debs
> > were reasonable in size.
> 
> The problem is that Isar wasn't applied on a larger scale yet. And it
> was only tested against the bitbake-unlike template pattern, instead of
> using layering consequently. We are doing both now, and the problems
> surface. All fixable, but the question is, where to do that best.

Ok, please let me know the priorities.


> >>> Claudius already suggested building custom .debs in Isar to solve that.
> > 
> > Yes. we built them where the effort was justified.
> 
> No, this is about making this feature available as library (class) so
> that you write three lines of recipe and have a customization deb
> package. That effort would be minimal for the user, and we only need to
> solve this once properly. If OE-core cannot provide that, we will have
> to implement it ourselves.

Ah, I see. This is also on TODO. This is an example what we could do for Debian
people, and this is what other tools don't provide.


> > As introduced above, Isar follows the Debian way to reuse as much as possible.
> > That is why just basing on OE won't make the tools work for the Isar layer OOTB
> > -- it will require the very adaptations we do now.
> 
> Can you provide a concrete code example? What all prevents using
> upstream wic (including its image class) more as-is and forces us to
> fork it?

For wic, it was mostly sysroot, plus a couple of bits that didn't work OOTB (no
idea whether that worked upstream). We don't have to fork, we need to adjust.
We just happen to stabilize it in the Isar repo before submitting upstream.


> >>> People that are coming from Yocto and want to switch to Isar will never
> >>> get their recipes to work, because all the utils-classes are missing in
> >>> Isar and would need to be ported/pulled-in.
> > 
> > This is what we ideally don't want to have. It is possible to build packages in
> > a basic OE way, but the preferred way is to debianize all packages. We want to
> > have stuff like clean, etc., but duplicating the whole OE functionality doesn't
> > make sense and won't work, since Isar and OE have different base systems and
> > toolchains.
> 
> Again, concrete examples would help to underline this high-level
> statement. We will be asked these questions over and over again, so we
> need the arguments ready at hand.

Till now, it's mostly toolchain and sysroot. The former should be overridable
via a var. The latter has to be adjusted.


> >>> Isar needs sudo, and someone has to integrate libpseudo.
> > 
> > This is on TODO.
> 
> It's actually WIP by us now - it's urgent because we need CI support,
> and that's in conflict with the sudo workarounds.

Please let us know when you start working on issues, to avoid effort
duplication.


> But, e.g., it has proper support for SRC_URI and fetching, proxies,
> mirrors, image types, and possibly more we need. Are all of these
> feature so tangled up with OE assumptions that stubbing and adjusting
> their dependencies would be worse than reimplementing them?

To my knowledge, SRC_URI and fetching work in bitbake and thus Isar. Wic image
types are also in Isar. I presume non-wic image types and proxies won't work
OOTB (due to sysroot and chroots, respectively) and would need to be adjusted.
Not sure about mirrors.


> > You can copy the recipe from Isar, and OE will be able to multistrap. The
> > problem is, you can't do anything useful with that afterwards due to the
> > different base system and compiler. OE != Debian, just like Ubuntu != Debian.
> 
> OK, that is a more concrete example to discuss: assumption of OE-core
> about the compiler and the build environment (when we are building, the
> exceptional case).

Building is not an exceptional case, every product has its custom applications.


> Where do they bite us when reusing OE-core to
> generate, e.g., images? Or fetching sources? Or any other case?

During image generation, it's sysroot (where stuff is located). Source fetching
is bitbake and should work in the same way.


> Yes, 90% of OE-core may be unused with Isar. But even 10% would still be
> an order of magnitude more lines of code than Isar has right now. That's
> the concern. If we have to port all that, or even just half of it over,
> we need strong reasons, ideally documented.

It isn't about the size, it's about architecture. OE-core is infrastructure +
metadata. If we go the OE way, the infrastructure should be split from the
metadata into different repos. But such kind of changes should not be suggested
by novices like me; the usual practice is to maintain them off-tree for several
years (e.g., PREEMPT-RT, etc.). Which in our case would mean forking and
maintaining OE-core till the upstream accepts it...

But while you are talking about percentage: What tools does OE provide? I'm
aware of isar-init-build-env, bitbake, wic, and runqemu. What else is there?


To summarize, what I hear:

+ Less effort to adjust tools for Debian in OE.

How I see it:

= The same effort to adjust tools for Debian in either repo.
- OE-core contains metadata that is not necessary.
- CM of OE-core metadata.
- Legal clearing of OE-core metadata.


I'll sync with Alex who can look at OE + Isar and report here.


With kind regards,
Baurzhan.

Re: [RFC] can we just extend openembedded to get Isar 2.0

[Jan Kiszka]

On 2017-07-25 00:53, Baurzhan Ismagulov wrote:
> On Fri, Jul 21, 2017 at 09:31:05AM +0200, Jan Kiszka wrote:
>>> The short answer is, Isar has started from scratch as a rootfs generator from
>>> existing binary distro + self-built packages. In plain German ;) , OE is not
>>> necessary for this purpose. OE is a distro, and Isar is a builder.
>>
>> OE-core is also a distro, but that is indeed the uninteresting part.
> 
> Not just uninteresting, it's 16 MiB of metadata that needs to be CM'ed and
> legally cleared.

That's not much, compared to what we actually install on the device.

I'm buying clearing arguments - as long as there are either real issues
(not cleanly licensed packages) or they are in line with technical
benefits. However, we at least have to process oe-core (via yocto)
frequently anyway, so there would be reuse.

> 
> 
>> We are interested in all the stuff in meta/classes, lib, you-name-it. All
>> what is still missing and needs to be copied, adjusted, and MAINTAINED
>> separately in isar so far.
> 
> Copying is not an issue. Adjusting is required in both ways. For maintenance,
> we already want to submit patches to Yocto / OE.

The fact is that the current classes have little to do with OE, thus
expose no reuse, just rewrites that still need additional work. Also,
Henning found a number of inconsistencies because - in contrast to
OE-core - we were not in lock-step with the bitbake we imported.

> 
> If one goes the OE layer way, the tools have to be adjusted anyway. But they
> won't be adjusted all at once. So, some tools would work, others would be
> present but won't work, creating a mess.

That adjustment needs are still unclear to me and probably require a
prototype to actually work out the details. At this level, discussion is
still too vague for me.

> 
> 
>> This gap is still to large, and we need to define a strategy now how to close
>> it sustainably.
> 
> The value of Isar is to provide convenient build environment for Debian-based
> systems. There are many things we can do on the Debian support side that no
> other tools do, and that should be the priority. Closing the gap with OE is
> going to be the same effort in either way.
> 
> 
>>> Of course, this is not that simple. You, and other people at various
>>> conferences, ask about the tools that you know and want to use, and this is
>>> exactly what we care about: Re-use mainstream tools and don't invent our own
>>> wheel.
> ...
>> The question is now, can we bend the OE-core towards building a Debian
>> distro in the Debian way with its pre-existing infrastructure, or do we
>> really have to fork it?
> 
> We don't fork it to have our own copies. We adjust them to provide the patches
> back upstream. So, for those tools, Isar is just a temporary storage place.
> 

Can you point to some OE or bitbake patches you have in staging?

> 
>>> So, to answer your original question, I don't exclude the possibility of using
>>> Isar on top of OE if it would ever make sense. However, keeping the possibility
>>> of using plain Isar is important for commercial product builders. Rationale:
>>> The short answer above, plus a major disadvantage: OE is huge, and this luggage
>>> is a problem for CM and integrators (I had to maintain OE in ClearCase, and I'm
>>> glad I don't anymore).
>>
>> OE-core is also just 58K LOC - smaller than bitbake itself, thus Isar.
> 
> I assume your LOC counting tool doesn't count 14 MiB of metadata?

"cloc ." - that's everything.

> 
> Bitbake is technically in the Isar repository just to ship the right version to
> any host version, but it isn't a part of Isar.
> 
> 
>> (And ClearCase is dead, just not buried everywhere ;).)
> 
> :) That was just an example that one has to CM the stuff.
> 
> 
>>> The problem with Debian support in OE is that it is very undebian. Debian has
>>> its strengths, and there are reasons why Debian developers strongly dislike OE
>>> (see e.g. [1], and I've heard many such discussions). Deby / meta-debian
>>> follows the OE way of building Debian packages, and currently considers moving
>>> to the Debian way. The reason is, as soon as you oeize a package, you have to
>>> maintain it. That is why Isar does things the Debian way and aims at avoiding
>>> massive package modifications; OE doesn't really help here.
>>
>> The idea is not to move to the OE way of doing Debian, for sure. It's
>> about adding the right way to the OE-core tools and libs that are useful
>> or even mandatory for Isar instead of forking or reimplementing them now.
> 
> The idea is clear, the answer is above: The adjustment effort is the same. We
> want to merge upstream, we don't reimplement.
> 

So far, we do reimplement (which includes code snippet imports).

> 
>>> Customization is implemented in configscript.sh. Some more structure could be
>>> introduced, but we didn't need it till now, as the changes beyond custom debs
>>> were reasonable in size.
>>
>> The problem is that Isar wasn't applied on a larger scale yet. And it
>> was only tested against the bitbake-unlike template pattern, instead of
>> using layering consequently. We are doing both now, and the problems
>> surface. All fixable, but the question is, where to do that best.
> 
> Ok, please let me know the priorities.
> 
> 
>>>>> Claudius already suggested building custom .debs in Isar to solve that.
>>>
>>> Yes. we built them where the effort was justified.
>>
>> No, this is about making this feature available as library (class) so
>> that you write three lines of recipe and have a customization deb
>> package. That effort would be minimal for the user, and we only need to
>> solve this once properly. If OE-core cannot provide that, we will have
>> to implement it ourselves.
> 
> Ah, I see. This is also on TODO. This is an example what we could do for Debian
> people, and this is what other tools don't provide.
> 

OK, great. Then the question is likely if we should extend or write a
separate tool for that or rather a bitbake recipe. Claudius was in favor
of a recipe, Henning found some tool in Debian.

> 
>>> As introduced above, Isar follows the Debian way to reuse as much as possible.
>>> That is why just basing on OE won't make the tools work for the Isar layer OOTB
>>> -- it will require the very adaptations we do now.
>>
>> Can you provide a concrete code example? What all prevents using
>> upstream wic (including its image class) more as-is and forces us to
>> fork it?
> 
> For wic, it was mostly sysroot, plus a couple of bits that didn't work OOTB (no
> idea whether that worked upstream). We don't have to fork, we need to adjust.
> We just happen to stabilize it in the Isar repo before submitting upstream.

Given that Isar != OE-core, you will end up with two patches on two
repos, and QA requires a second stage (first in Isar, then in OE-core).
That's why using a common code base would be beneficial as well.

> 
> 
>>>>> People that are coming from Yocto and want to switch to Isar will never
>>>>> get their recipes to work, because all the utils-classes are missing in
>>>>> Isar and would need to be ported/pulled-in.
>>>
>>> This is what we ideally don't want to have. It is possible to build packages in
>>> a basic OE way, but the preferred way is to debianize all packages. We want to
>>> have stuff like clean, etc., but duplicating the whole OE functionality doesn't
>>> make sense and won't work, since Isar and OE have different base systems and
>>> toolchains.
>>
>> Again, concrete examples would help to underline this high-level
>> statement. We will be asked these questions over and over again, so we
>> need the arguments ready at hand.
> 
> Till now, it's mostly toolchain and sysroot. The former should be overridable
> via a var. The latter has to be adjusted.
> 

Example at hand?

> 
>>>>> Isar needs sudo, and someone has to integrate libpseudo.
>>>
>>> This is on TODO.
>>
>> It's actually WIP by us now - it's urgent because we need CI support,
>> and that's in conflict with the sudo workarounds.
> 
> Please let us know when you start working on issues, to avoid effort
> duplication.

"It's actually WIP by us *now*". Once we have a prototype, it will be
shared on the list.

> 
> 
>> But, e.g., it has proper support for SRC_URI and fetching, proxies,
>> mirrors, image types, and possibly more we need. Are all of these
>> feature so tangled up with OE assumptions that stubbing and adjusting
>> their dependencies would be worse than reimplementing them?
> 
> To my knowledge, SRC_URI and fetching work in bitbake and thus Isar.

Nope, I tried it. It's only available for the dpkg class, not for all
classes.

> Wic image types are also in Isar.

Nope, otherwise you could build in a single step, without an explicit
wic call.

> I presume non-wic image types and proxies won't work

They weren't copied or forked yet, right.

> OOTB (due to sysroot and chroots, respectively) and would need to be adjusted.

Not if we followed the same layout that OE-core generates. I mean, after
creating the rootfs folder, we are on the same page again, and that's
where all the image types start off.

> Not sure about mirrors.

If it wasn't tested yet, it's broken for sure. E.g., quite a few
variables are not initialized yet that the OE-core classes and even
bitbakes expect.

> 
> 
>>> You can copy the recipe from Isar, and OE will be able to multistrap. The
>>> problem is, you can't do anything useful with that afterwards due to the
>>> different base system and compiler. OE != Debian, just like Ubuntu != Debian.
>>
>> OK, that is a more concrete example to discuss: assumption of OE-core
>> about the compiler and the build environment (when we are building, the
>> exceptional case).
> 
> Building is not an exceptional case, every product has its custom applications.
> 

Compared to the primarily goal, installing tons of prebuilt packages,
it's exceptional to Isar. That's what I meant. I agree that we do need
to build a handful of packages in most projects as well.

> 
>> Where do they bite us when reusing OE-core to
>> generate, e.g., images? Or fetching sources? Or any other case?
> 
> During image generation, it's sysroot (where stuff is located). Source fetching
> is bitbake and should work in the same way.
> 

Fetching - like many things bitbake does - is controlled by a bunch of
variables in the classes and recipes. Due to the differences there,
things do not work as expected.

> 
>> Yes, 90% of OE-core may be unused with Isar. But even 10% would still be
>> an order of magnitude more lines of code than Isar has right now. That's
>> the concern. If we have to port all that, or even just half of it over,
>> we need strong reasons, ideally documented.
> 
> It isn't about the size, it's about architecture. OE-core is infrastructure +
> metadata. If we go the OE way, the infrastructure should be split from the
> metadata into different repos. But such kind of changes should not be suggested
> by novices like me; the usual practice is to maintain them off-tree for several
> years (e.g., PREEMPT-RT, etc.). Which in our case would mean forking and
> maintaining OE-core till the upstream accepts it...

metadata could still be split off later. I really don't care about it
now as we are using it anyway for Yocto in other projects. I agree that
would be be nicer to have the core free of package recipes and patches
we don't use. But that's not a hard precondition.

> 
> But while you are talking about percentage: What tools does OE provide? I'm
> aware of isar-init-build-env, bitbake, wic, and runqemu. What else is there?
> 

Classes, the secret source to glue all that together and make it
conveniently usable.

> 
> To summarize, what I hear:
> 
> + Less effort to adjust tools for Debian in OE.
> 
> How I see it:
> 
> = The same effort to adjust tools for Debian in either repo.
> - OE-core contains metadata that is not necessary.

Unneeded, but acceptable for now.

> - CM of OE-core metadata.

Again, CM is not a compelling reason. We are dealing with packages in
*every* embedded Linux device which are at least one order of magnitude
larger than OE-core.

> - Legal clearing of OE-core metadata.

See above: already done.

We need more technical reasons.

> 
> 
> I'll sync with Alex who can look at OE + Isar and report here.

Let's discuss this in our local round. We may contribute to that as well.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [RFC] can we just extend openembedded to get Isar 2.0

[Claudius Heine]

Hi,

On 07/25/2017 08:08 AM, [ext] Jan Kiszka wrote:
> On 2017-07-25 00:53, Baurzhan Ismagulov wrote:
>> On Fri, Jul 21, 2017 at 09:31:05AM +0200, Jan Kiszka wrote:
>>>>>> Claudius already suggested building custom .debs in Isar to solve that.
>>>>
>>>> Yes. we built them where the effort was justified.
>>>
>>> No, this is about making this feature available as library (class) so
>>> that you write three lines of recipe and have a customization deb
>>> package. That effort would be minimal for the user, and we only need to
>>> solve this once properly. If OE-core cannot provide that, we will have
>>> to implement it ourselves.
>>
>> Ah, I see. This is also on TODO. This is an example what we could do for Debian
>> people, and this is what other tools don't provide.
>>
> 
> OK, great. Then the question is likely if we should extend or write a
> separate tool for that or rather a bitbake recipe. Claudius was in favor
> of a recipe, Henning found some tool in Debian.

I am more in favor of a bbclass for this. Creating simple packages this 
way should be easy compared to expressing runtime and build dependencies 
to upstream packages or own packages created by a bitbake recipe and 
resolving and installing them into the build chroot beforehand.
It would be great to get PACKAGES, RDEPENDS_${PN}, FILES_${PN}, etc. 
working again.

I am not sure of the best way to do these kind of tasks in isar.

To integrate isar into OE one way could be to have a recipe that fetches 
the packet index from the repository and populates the bitbake datastore 
with recipes or packages for every package in the repository at runtime. 
Or, if that is too hacky or does not scale well, hook into the 'DEPENDS' 
resolve mechansim and look into the debian package index there and 
generate recipes/packages into the datastore just for the requested 
packages.
Also integrate use qemu + native debian toolchain as alternative cross 
compiler. This might need some additional finagling.

>>>> You can copy the recipe from Isar, and OE will be able to multistrap. The
>>>> problem is, you can't do anything useful with that afterwards due to the
>>>> different base system and compiler. OE != Debian, just like Ubuntu != Debian.
>>>
>>> OK, that is a more concrete example to discuss: assumption of OE-core
>>> about the compiler and the build environment (when we are building, the
>>> exceptional case).
>>
>> Building is not an exceptional case, every product has its custom applications.
>>
> 
> Compared to the primarily goal, installing tons of prebuilt packages,
> it's exceptional to Isar. That's what I meant. I agree that we do need
> to build a handful of packages in most projects as well.

Maybe I am captain obvious here, but should be possible to build static 
binaries with muslc in oe. Those would be independent of the toolchain.

>> But while you are talking about percentage: What tools does OE provide? I'm
>> aware of isar-init-build-env, bitbake, wic, and runqemu. What else is there?
>>
> 
> Classes, the secret source to glue all that together and make it
> conveniently usable.
> 
>>
>> To summarize, what I hear:
>>
>> + Less effort to adjust tools for Debian in OE.

Some of them might be a bit too hopeful and based on my implementation 
proposal:

+ Less intrusive change to the build system (bitbake + oe)
   -> Less work to keep it working with upstream
   -> Higher chance of beeing integrated into upstream
   -> Resolve many regressions of isar (compared to bitbake + oe)
   -> No need to reimplement most of the functionality (creating 
packages, images)
   -> Mostly compatible with oe recipes

Cheers,
Claudius

Re: [RFC] can we just extend openembedded to get Isar 2.0

[Kazuhiro Hayashi]

Hello,

>>>>> The problem with Debian support in OE is that it is very undebian. Debian has
>>>>> its strengths, and there are reasons why Debian developers strongly dislike OE
>>>>> (see e.g. [1], and I've heard many such discussions). Deby / meta-debian
>>>>> follows the OE way of building Debian packages, and currently considers moving
>>>>> to the Debian way. The reason is, as soon as you oeize a package, you have to
>>>>> maintain it. That is why Isar does things the Debian way and aims at avoiding
>>>>> massive package modifications; OE doesn't really help here.
>>>>
>>>> The idea is not to move to the OE way of doing Debian, for sure. It's
>>>> about adding the right way to the OE-core tools and libs that are useful
>>>> or even mandatory for Isar instead of forking or reimplementing them now.
>>>
>>> The idea is clear, the answer is above: The adjustment effort is the same. We
>>> want to merge upstream, we don't reimplement.
>>
>> So far, we do reimplement (which includes code snippet imports).

In Deby, there are more than 700 recipes (.bb) and all of them are
implemented following OE-Core way. As Baurzhan mentioned, the main issue
is to create and maintain such recipes ourselves. On the other hands,
as an advantage, the recipes can reuse (inherits) most common stuffs
like .bbclass, .conf, etc. provided by OE-Core without any changes.

Examples:
* Tasks for cross-building own kernel (kernel.bbclass)
* Functions to run menuconfig through bitbake (cml1.bbclass)
* ptest infrastructure (ptest.bbclass)
* Packaging functions (package_deb.bbclass)
* Recipes for generating rootfs and SDK
   (core-image-minimal.bb, meta-toolchain.bb)
* sstate caching (sstate.bbclass)

IIUC, most .bbclass depend on OE-Core infrastructure (some of them
depend on metadata as well). For examples, they add several recipes
(especially native recipes) provided in OE-Core to their DEPENDS,
assume the default task flow (do_configure, do_compile, etc.),
cross toolchain and sysroots of OE-Core. To split such OE-Core
dependent stuffs from existing .bbclass (or to adjust them for
other system like Isar) may require much effort.

Additionally, big issues we met are:
* There is no efficient way to append .bbclass
* Specification of such kind of common stuffs frequently change in upstream

Regarding libraries (meta/lib), I've never tried to reuse or
inherit classes or functions in them, but I guess that they have
less dependencies on OE-Core infrastructure and metadata than .bbclass.
Some of them get values from several global variables define in
the default configs (.conf) in OE-Core.
Regarding scripts, we are just users, no experience of customization.

As Baurzhan mentioned, Deby is now considering moving to the Debian way,
but this change affects only how to implement .bb for Debian packages.
In other words, we will keep using OE-Core infrastructure as before
under the following policies.
* Manage all files (.bb, .bbclass, .conf, etc.) in an independent layer
   (meta-debian.git) which works on top of OE-Core
* Don't modify (fork) OE-Core layer, instead, adjust by overriding variables

Actually, due to the above reasons, there might not be so many existing
stuffs, which can be shared with Debian way, in OE-Core (except bitbake).
At least, supporting sysroot and adapting Debian toolchain to OE-Core
are required to reuse most existing .bbclass. We are just considering
the better solution to implement Debian way on top of OE-Core
infrastructure, including such wide adaptation.

Best regards,
Kazuhiro

-- 
  Kazuhiro Hayashi
  E-mail: kazuhiro3.hayashi@toshiba.co.jp