Re: [RFC] can we just extend openembedded to get Isar 2.0

[Jan Kiszka <jan.kiszka@siemens.com>]

On 2017-07-25 00:53, Baurzhan Ismagulov wrote:
> On Fri, Jul 21, 2017 at 09:31:05AM +0200, Jan Kiszka wrote:
>>> The short answer is, Isar has started from scratch as a rootfs generator from
>>> existing binary distro + self-built packages. In plain German ;) , OE is not
>>> necessary for this purpose. OE is a distro, and Isar is a builder.
>>
>> OE-core is also a distro, but that is indeed the uninteresting part.
> 
> Not just uninteresting, it's 16 MiB of metadata that needs to be CM'ed and
> legally cleared.

That's not much, compared to what we actually install on the device.

I'm buying clearing arguments - as long as there are either real issues
(not cleanly licensed packages) or they are in line with technical
benefits. However, we at least have to process oe-core (via yocto)
frequently anyway, so there would be reuse.

> 
> 
>> We are interested in all the stuff in meta/classes, lib, you-name-it. All
>> what is still missing and needs to be copied, adjusted, and MAINTAINED
>> separately in isar so far.
> 
> Copying is not an issue. Adjusting is required in both ways. For maintenance,
> we already want to submit patches to Yocto / OE.

The fact is that the current classes have little to do with OE, thus
expose no reuse, just rewrites that still need additional work. Also,
Henning found a number of inconsistencies because - in contrast to
OE-core - we were not in lock-step with the bitbake we imported.

> 
> If one goes the OE layer way, the tools have to be adjusted anyway. But they
> won't be adjusted all at once. So, some tools would work, others would be
> present but won't work, creating a mess.

That adjustment needs are still unclear to me and probably require a
prototype to actually work out the details. At this level, discussion is
still too vague for me.

> 
> 
>> This gap is still to large, and we need to define a strategy now how to close
>> it sustainably.
> 
> The value of Isar is to provide convenient build environment for Debian-based
> systems. There are many things we can do on the Debian support side that no
> other tools do, and that should be the priority. Closing the gap with OE is
> going to be the same effort in either way.
> 
> 
>>> Of course, this is not that simple. You, and other people at various
>>> conferences, ask about the tools that you know and want to use, and this is
>>> exactly what we care about: Re-use mainstream tools and don't invent our own
>>> wheel.
> ...
>> The question is now, can we bend the OE-core towards building a Debian
>> distro in the Debian way with its pre-existing infrastructure, or do we
>> really have to fork it?
> 
> We don't fork it to have our own copies. We adjust them to provide the patches
> back upstream. So, for those tools, Isar is just a temporary storage place.
> 

Can you point to some OE or bitbake patches you have in staging?

> 
>>> So, to answer your original question, I don't exclude the possibility of using
>>> Isar on top of OE if it would ever make sense. However, keeping the possibility
>>> of using plain Isar is important for commercial product builders. Rationale:
>>> The short answer above, plus a major disadvantage: OE is huge, and this luggage
>>> is a problem for CM and integrators (I had to maintain OE in ClearCase, and I'm
>>> glad I don't anymore).
>>
>> OE-core is also just 58K LOC - smaller than bitbake itself, thus Isar.
> 
> I assume your LOC counting tool doesn't count 14 MiB of metadata?

"cloc ." - that's everything.

> 
> Bitbake is technically in the Isar repository just to ship the right version to
> any host version, but it isn't a part of Isar.
> 
> 
>> (And ClearCase is dead, just not buried everywhere ;).)
> 
> :) That was just an example that one has to CM the stuff.
> 
> 
>>> The problem with Debian support in OE is that it is very undebian. Debian has
>>> its strengths, and there are reasons why Debian developers strongly dislike OE
>>> (see e.g. [1], and I've heard many such discussions). Deby / meta-debian
>>> follows the OE way of building Debian packages, and currently considers moving
>>> to the Debian way. The reason is, as soon as you oeize a package, you have to
>>> maintain it. That is why Isar does things the Debian way and aims at avoiding
>>> massive package modifications; OE doesn't really help here.
>>
>> The idea is not to move to the OE way of doing Debian, for sure. It's
>> about adding the right way to the OE-core tools and libs that are useful
>> or even mandatory for Isar instead of forking or reimplementing them now.
> 
> The idea is clear, the answer is above: The adjustment effort is the same. We
> want to merge upstream, we don't reimplement.
> 

So far, we do reimplement (which includes code snippet imports).

> 
>>> Customization is implemented in configscript.sh. Some more structure could be
>>> introduced, but we didn't need it till now, as the changes beyond custom debs
>>> were reasonable in size.
>>
>> The problem is that Isar wasn't applied on a larger scale yet. And it
>> was only tested against the bitbake-unlike template pattern, instead of
>> using layering consequently. We are doing both now, and the problems
>> surface. All fixable, but the question is, where to do that best.
> 
> Ok, please let me know the priorities.
> 
> 
>>>>> Claudius already suggested building custom .debs in Isar to solve that.
>>>
>>> Yes. we built them where the effort was justified.
>>
>> No, this is about making this feature available as library (class) so
>> that you write three lines of recipe and have a customization deb
>> package. That effort would be minimal for the user, and we only need to
>> solve this once properly. If OE-core cannot provide that, we will have
>> to implement it ourselves.
> 
> Ah, I see. This is also on TODO. This is an example what we could do for Debian
> people, and this is what other tools don't provide.
> 

OK, great. Then the question is likely if we should extend or write a
separate tool for that or rather a bitbake recipe. Claudius was in favor
of a recipe, Henning found some tool in Debian.

> 
>>> As introduced above, Isar follows the Debian way to reuse as much as possible.
>>> That is why just basing on OE won't make the tools work for the Isar layer OOTB
>>> -- it will require the very adaptations we do now.
>>
>> Can you provide a concrete code example? What all prevents using
>> upstream wic (including its image class) more as-is and forces us to
>> fork it?
> 
> For wic, it was mostly sysroot, plus a couple of bits that didn't work OOTB (no
> idea whether that worked upstream). We don't have to fork, we need to adjust.
> We just happen to stabilize it in the Isar repo before submitting upstream.

Given that Isar != OE-core, you will end up with two patches on two
repos, and QA requires a second stage (first in Isar, then in OE-core).
That's why using a common code base would be beneficial as well.

> 
> 
>>>>> People that are coming from Yocto and want to switch to Isar will never
>>>>> get their recipes to work, because all the utils-classes are missing in
>>>>> Isar and would need to be ported/pulled-in.
>>>
>>> This is what we ideally don't want to have. It is possible to build packages in
>>> a basic OE way, but the preferred way is to debianize all packages. We want to
>>> have stuff like clean, etc., but duplicating the whole OE functionality doesn't
>>> make sense and won't work, since Isar and OE have different base systems and
>>> toolchains.
>>
>> Again, concrete examples would help to underline this high-level
>> statement. We will be asked these questions over and over again, so we
>> need the arguments ready at hand.
> 
> Till now, it's mostly toolchain and sysroot. The former should be overridable
> via a var. The latter has to be adjusted.
> 

Example at hand?

> 
>>>>> Isar needs sudo, and someone has to integrate libpseudo.
>>>
>>> This is on TODO.
>>
>> It's actually WIP by us now - it's urgent because we need CI support,
>> and that's in conflict with the sudo workarounds.
> 
> Please let us know when you start working on issues, to avoid effort
> duplication.

"It's actually WIP by us *now*". Once we have a prototype, it will be
shared on the list.

> 
> 
>> But, e.g., it has proper support for SRC_URI and fetching, proxies,
>> mirrors, image types, and possibly more we need. Are all of these
>> feature so tangled up with OE assumptions that stubbing and adjusting
>> their dependencies would be worse than reimplementing them?
> 
> To my knowledge, SRC_URI and fetching work in bitbake and thus Isar.

Nope, I tried it. It's only available for the dpkg class, not for all
classes.

> Wic image types are also in Isar.

Nope, otherwise you could build in a single step, without an explicit
wic call.

> I presume non-wic image types and proxies won't work

They weren't copied or forked yet, right.

> OOTB (due to sysroot and chroots, respectively) and would need to be adjusted.

Not if we followed the same layout that OE-core generates. I mean, after
creating the rootfs folder, we are on the same page again, and that's
where all the image types start off.

> Not sure about mirrors.

If it wasn't tested yet, it's broken for sure. E.g., quite a few
variables are not initialized yet that the OE-core classes and even
bitbakes expect.

> 
> 
>>> You can copy the recipe from Isar, and OE will be able to multistrap. The
>>> problem is, you can't do anything useful with that afterwards due to the
>>> different base system and compiler. OE != Debian, just like Ubuntu != Debian.
>>
>> OK, that is a more concrete example to discuss: assumption of OE-core
>> about the compiler and the build environment (when we are building, the
>> exceptional case).
> 
> Building is not an exceptional case, every product has its custom applications.
> 

Compared to the primarily goal, installing tons of prebuilt packages,
it's exceptional to Isar. That's what I meant. I agree that we do need
to build a handful of packages in most projects as well.

> 
>> Where do they bite us when reusing OE-core to
>> generate, e.g., images? Or fetching sources? Or any other case?
> 
> During image generation, it's sysroot (where stuff is located). Source fetching
> is bitbake and should work in the same way.
> 

Fetching - like many things bitbake does - is controlled by a bunch of
variables in the classes and recipes. Due to the differences there,
things do not work as expected.

> 
>> Yes, 90% of OE-core may be unused with Isar. But even 10% would still be
>> an order of magnitude more lines of code than Isar has right now. That's
>> the concern. If we have to port all that, or even just half of it over,
>> we need strong reasons, ideally documented.
> 
> It isn't about the size, it's about architecture. OE-core is infrastructure +
> metadata. If we go the OE way, the infrastructure should be split from the
> metadata into different repos. But such kind of changes should not be suggested
> by novices like me; the usual practice is to maintain them off-tree for several
> years (e.g., PREEMPT-RT, etc.). Which in our case would mean forking and
> maintaining OE-core till the upstream accepts it...

metadata could still be split off later. I really don't care about it
now as we are using it anyway for Yocto in other projects. I agree that
would be be nicer to have the core free of package recipes and patches
we don't use. But that's not a hard precondition.

> 
> But while you are talking about percentage: What tools does OE provide? I'm
> aware of isar-init-build-env, bitbake, wic, and runqemu. What else is there?
> 

Classes, the secret source to glue all that together and make it
conveniently usable.

> 
> To summarize, what I hear:
> 
> + Less effort to adjust tools for Debian in OE.
> 
> How I see it:
> 
> = The same effort to adjust tools for Debian in either repo.
> - OE-core contains metadata that is not necessary.

Unneeded, but acceptable for now.

> - CM of OE-core metadata.

Again, CM is not a compelling reason. We are dealing with packages in
*every* embedded Linux device which are at least one order of magnitude
larger than OE-core.

> - Legal clearing of OE-core metadata.

See above: already done.

We need more technical reasons.

> 
> 
> I'll sync with Alex who can look at OE + Isar and report here.

Let's discuss this in our local round. We may contribute to that as well.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux