From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6444767352800149504 X-Received: by 10.25.193.77 with SMTP id r74mr1053924lff.14.1500962925749; Mon, 24 Jul 2017 23:08:45 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.46.21.12 with SMTP id s12ls1490424ljd.22.gmail; Mon, 24 Jul 2017 23:08:45 -0700 (PDT) X-Received: by 10.46.5.9 with SMTP id 9mr3473674ljf.28.1500962925318; Mon, 24 Jul 2017 23:08:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1500962925; cv=none; d=google.com; s=arc-20160816; b=u4nfX5c4tZw6NIKkS3WqqnvWzOSwVXINAEvmuESOrIhXm91Kba7dFdtn5Djb+ZcvTV MlsQoV+6TJI5SuDCtp7HZwiCzEP5HEhMQQckJutNvE4BdThLIFC3Uy6ntydxUrM7RIQt JQ1yvKDQy2TIolgSANVQuDbpwvfWYTS1lNQtPThJiM3YcH+xs7KYHBK/JoANmHILq2jL jdgldTCj5iCrx7adccjPiInkeZFNI1pQ70m4xFjrDZ+BVn3TQ4PMLoENfXNMPm5JO1Ox dSrN0Xq/bP7yKpufcMumN8tNQ1Ny0XFes7XRPOkWV8fH+NgO/I6xpcX3jaqiB7uf3D+F BWIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :arc-authentication-results; bh=E1/vLCoYAM5c1+jcptBCx9sXkuORv7ztJYWlYCqmWR0=; b=wFRorm2AbMu0AA5nW0KT4DVWw4s9obA7Z5RmaDHZPtqlsC7HDQ+STV8mms0qhOa53w MPOLw+qxjRiWSARU/WEw9RSx6d4Dx5wmvuuFka6bCgZFUyqrUTH2w7o/pdOsTFW4+K/G GImARvw/7JiaM+n0zROSoxsv8Ev+jaeboJ+aFaOXxIhRMQ8EUqV5qEm/F/2bQQb6Oh7d a4FXSEDiCkImXFTvkxpWxNFlDC4flZ1hW7jWB1slSra5g9zmUQ31qdO+FR8KAsr1SeVj Za99XJkhjGVsCmYH17XAVx2pN9eEvUvpp35ddPW1iE2rdLg282kHfDiDfhaaMpNoehO8 pCKw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 192.35.17.14 is neither permitted nor denied by best guess record for domain of jan.kiszka@siemens.com) smtp.mailfrom=jan.kiszka@siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id m191si2622859wmg.9.2017.07.24.23.08.45 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 24 Jul 2017 23:08:45 -0700 (PDT) Received-SPF: neutral (google.com: 192.35.17.14 is neither permitted nor denied by best guess record for domain of jan.kiszka@siemens.com) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 192.35.17.14 is neither permitted nor denied by best guess record for domain of jan.kiszka@siemens.com) smtp.mailfrom=jan.kiszka@siemens.com Received: from mail1.siemens.de (mail1.siemens.de [139.23.33.14]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id v6P68hL8012381 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 25 Jul 2017 08:08:44 +0200 Received: from md1f2u6c.ww002.siemens.net ([139.22.124.204]) by mail1.siemens.de (8.15.2/8.15.2) with ESMTP id v6P68gFD015089; Tue, 25 Jul 2017 08:08:43 +0200 Subject: Re: [RFC] can we just extend openembedded to get Isar 2.0 To: isar-users@googlegroups.com, Henning Schild , KOBAYASHI Yoshitake , Daniel Sangorrin , kazuhiro3.hayashi@toshiba.co.jp, asmirnov@ilbers.de, Claudius Heine References: <20170720103037.794aab2e@md1em3qc> <20170720230529.GB5040@yssyq.radix50.net> <844d35a9-6ffd-9a84-b324-21acec937eaf@siemens.com> <20170724225350.GC3612@yssyq.radix50.net> From: Jan Kiszka Message-ID: Date: Tue, 25 Jul 2017 08:08:42 +0200 User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.8.1.12) Gecko/20080226 SUSE/2.0.0.12-1.1 Thunderbird/2.0.0.12 Mnenhy/0.7.5.666 MIME-Version: 1.0 In-Reply-To: <20170724225350.GC3612@yssyq.radix50.net> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: AiYNl7LGOxz8 On 2017-07-25 00:53, Baurzhan Ismagulov wrote: > On Fri, Jul 21, 2017 at 09:31:05AM +0200, Jan Kiszka wrote: >>> The short answer is, Isar has started from scratch as a rootfs generator from >>> existing binary distro + self-built packages. In plain German ;) , OE is not >>> necessary for this purpose. OE is a distro, and Isar is a builder. >> >> OE-core is also a distro, but that is indeed the uninteresting part. > > Not just uninteresting, it's 16 MiB of metadata that needs to be CM'ed and > legally cleared. That's not much, compared to what we actually install on the device. I'm buying clearing arguments - as long as there are either real issues (not cleanly licensed packages) or they are in line with technical benefits. However, we at least have to process oe-core (via yocto) frequently anyway, so there would be reuse. > > >> We are interested in all the stuff in meta/classes, lib, you-name-it. All >> what is still missing and needs to be copied, adjusted, and MAINTAINED >> separately in isar so far. > > Copying is not an issue. Adjusting is required in both ways. For maintenance, > we already want to submit patches to Yocto / OE. The fact is that the current classes have little to do with OE, thus expose no reuse, just rewrites that still need additional work. Also, Henning found a number of inconsistencies because - in contrast to OE-core - we were not in lock-step with the bitbake we imported. > > If one goes the OE layer way, the tools have to be adjusted anyway. But they > won't be adjusted all at once. So, some tools would work, others would be > present but won't work, creating a mess. That adjustment needs are still unclear to me and probably require a prototype to actually work out the details. At this level, discussion is still too vague for me. > > >> This gap is still to large, and we need to define a strategy now how to close >> it sustainably. > > The value of Isar is to provide convenient build environment for Debian-based > systems. There are many things we can do on the Debian support side that no > other tools do, and that should be the priority. Closing the gap with OE is > going to be the same effort in either way. > > >>> Of course, this is not that simple. You, and other people at various >>> conferences, ask about the tools that you know and want to use, and this is >>> exactly what we care about: Re-use mainstream tools and don't invent our own >>> wheel. > ... >> The question is now, can we bend the OE-core towards building a Debian >> distro in the Debian way with its pre-existing infrastructure, or do we >> really have to fork it? > > We don't fork it to have our own copies. We adjust them to provide the patches > back upstream. So, for those tools, Isar is just a temporary storage place. > Can you point to some OE or bitbake patches you have in staging? > >>> So, to answer your original question, I don't exclude the possibility of using >>> Isar on top of OE if it would ever make sense. However, keeping the possibility >>> of using plain Isar is important for commercial product builders. Rationale: >>> The short answer above, plus a major disadvantage: OE is huge, and this luggage >>> is a problem for CM and integrators (I had to maintain OE in ClearCase, and I'm >>> glad I don't anymore). >> >> OE-core is also just 58K LOC - smaller than bitbake itself, thus Isar. > > I assume your LOC counting tool doesn't count 14 MiB of metadata? "cloc ." - that's everything. > > Bitbake is technically in the Isar repository just to ship the right version to > any host version, but it isn't a part of Isar. > > >> (And ClearCase is dead, just not buried everywhere ;).) > > :) That was just an example that one has to CM the stuff. > > >>> The problem with Debian support in OE is that it is very undebian. Debian has >>> its strengths, and there are reasons why Debian developers strongly dislike OE >>> (see e.g. [1], and I've heard many such discussions). Deby / meta-debian >>> follows the OE way of building Debian packages, and currently considers moving >>> to the Debian way. The reason is, as soon as you oeize a package, you have to >>> maintain it. That is why Isar does things the Debian way and aims at avoiding >>> massive package modifications; OE doesn't really help here. >> >> The idea is not to move to the OE way of doing Debian, for sure. It's >> about adding the right way to the OE-core tools and libs that are useful >> or even mandatory for Isar instead of forking or reimplementing them now. > > The idea is clear, the answer is above: The adjustment effort is the same. We > want to merge upstream, we don't reimplement. > So far, we do reimplement (which includes code snippet imports). > >>> Customization is implemented in configscript.sh. Some more structure could be >>> introduced, but we didn't need it till now, as the changes beyond custom debs >>> were reasonable in size. >> >> The problem is that Isar wasn't applied on a larger scale yet. And it >> was only tested against the bitbake-unlike template pattern, instead of >> using layering consequently. We are doing both now, and the problems >> surface. All fixable, but the question is, where to do that best. > > Ok, please let me know the priorities. > > >>>>> Claudius already suggested building custom .debs in Isar to solve that. >>> >>> Yes. we built them where the effort was justified. >> >> No, this is about making this feature available as library (class) so >> that you write three lines of recipe and have a customization deb >> package. That effort would be minimal for the user, and we only need to >> solve this once properly. If OE-core cannot provide that, we will have >> to implement it ourselves. > > Ah, I see. This is also on TODO. This is an example what we could do for Debian > people, and this is what other tools don't provide. > OK, great. Then the question is likely if we should extend or write a separate tool for that or rather a bitbake recipe. Claudius was in favor of a recipe, Henning found some tool in Debian. > >>> As introduced above, Isar follows the Debian way to reuse as much as possible. >>> That is why just basing on OE won't make the tools work for the Isar layer OOTB >>> -- it will require the very adaptations we do now. >> >> Can you provide a concrete code example? What all prevents using >> upstream wic (including its image class) more as-is and forces us to >> fork it? > > For wic, it was mostly sysroot, plus a couple of bits that didn't work OOTB (no > idea whether that worked upstream). We don't have to fork, we need to adjust. > We just happen to stabilize it in the Isar repo before submitting upstream. Given that Isar != OE-core, you will end up with two patches on two repos, and QA requires a second stage (first in Isar, then in OE-core). That's why using a common code base would be beneficial as well. > > >>>>> People that are coming from Yocto and want to switch to Isar will never >>>>> get their recipes to work, because all the utils-classes are missing in >>>>> Isar and would need to be ported/pulled-in. >>> >>> This is what we ideally don't want to have. It is possible to build packages in >>> a basic OE way, but the preferred way is to debianize all packages. We want to >>> have stuff like clean, etc., but duplicating the whole OE functionality doesn't >>> make sense and won't work, since Isar and OE have different base systems and >>> toolchains. >> >> Again, concrete examples would help to underline this high-level >> statement. We will be asked these questions over and over again, so we >> need the arguments ready at hand. > > Till now, it's mostly toolchain and sysroot. The former should be overridable > via a var. The latter has to be adjusted. > Example at hand? > >>>>> Isar needs sudo, and someone has to integrate libpseudo. >>> >>> This is on TODO. >> >> It's actually WIP by us now - it's urgent because we need CI support, >> and that's in conflict with the sudo workarounds. > > Please let us know when you start working on issues, to avoid effort > duplication. "It's actually WIP by us *now*". Once we have a prototype, it will be shared on the list. > > >> But, e.g., it has proper support for SRC_URI and fetching, proxies, >> mirrors, image types, and possibly more we need. Are all of these >> feature so tangled up with OE assumptions that stubbing and adjusting >> their dependencies would be worse than reimplementing them? > > To my knowledge, SRC_URI and fetching work in bitbake and thus Isar. Nope, I tried it. It's only available for the dpkg class, not for all classes. > Wic image types are also in Isar. Nope, otherwise you could build in a single step, without an explicit wic call. > I presume non-wic image types and proxies won't work They weren't copied or forked yet, right. > OOTB (due to sysroot and chroots, respectively) and would need to be adjusted. Not if we followed the same layout that OE-core generates. I mean, after creating the rootfs folder, we are on the same page again, and that's where all the image types start off. > Not sure about mirrors. If it wasn't tested yet, it's broken for sure. E.g., quite a few variables are not initialized yet that the OE-core classes and even bitbakes expect. > > >>> You can copy the recipe from Isar, and OE will be able to multistrap. The >>> problem is, you can't do anything useful with that afterwards due to the >>> different base system and compiler. OE != Debian, just like Ubuntu != Debian. >> >> OK, that is a more concrete example to discuss: assumption of OE-core >> about the compiler and the build environment (when we are building, the >> exceptional case). > > Building is not an exceptional case, every product has its custom applications. > Compared to the primarily goal, installing tons of prebuilt packages, it's exceptional to Isar. That's what I meant. I agree that we do need to build a handful of packages in most projects as well. > >> Where do they bite us when reusing OE-core to >> generate, e.g., images? Or fetching sources? Or any other case? > > During image generation, it's sysroot (where stuff is located). Source fetching > is bitbake and should work in the same way. > Fetching - like many things bitbake does - is controlled by a bunch of variables in the classes and recipes. Due to the differences there, things do not work as expected. > >> Yes, 90% of OE-core may be unused with Isar. But even 10% would still be >> an order of magnitude more lines of code than Isar has right now. That's >> the concern. If we have to port all that, or even just half of it over, >> we need strong reasons, ideally documented. > > It isn't about the size, it's about architecture. OE-core is infrastructure + > metadata. If we go the OE way, the infrastructure should be split from the > metadata into different repos. But such kind of changes should not be suggested > by novices like me; the usual practice is to maintain them off-tree for several > years (e.g., PREEMPT-RT, etc.). Which in our case would mean forking and > maintaining OE-core till the upstream accepts it... metadata could still be split off later. I really don't care about it now as we are using it anyway for Yocto in other projects. I agree that would be be nicer to have the core free of package recipes and patches we don't use. But that's not a hard precondition. > > But while you are talking about percentage: What tools does OE provide? I'm > aware of isar-init-build-env, bitbake, wic, and runqemu. What else is there? > Classes, the secret source to glue all that together and make it conveniently usable. > > To summarize, what I hear: > > + Less effort to adjust tools for Debian in OE. > > How I see it: > > = The same effort to adjust tools for Debian in either repo. > - OE-core contains metadata that is not necessary. Unneeded, but acceptable for now. > - CM of OE-core metadata. Again, CM is not a compelling reason. We are dealing with packages in *every* embedded Linux device which are at least one order of magnitude larger than OE-core. > - Legal clearing of OE-core metadata. See above: already done. We need more technical reasons. > > > I'll sync with Alex who can look at OE + Isar and report here. Let's discuss this in our local round. We may contribute to that as well. Jan -- Siemens AG, Corporate Technology, CT RDA ITP SES-DE Corporate Competence Center Embedded Linux