From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6444767352800149504 X-Received: by 10.28.111.200 with SMTP id c69mr421954wmi.27.1500622267125; Fri, 21 Jul 2017 00:31:07 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.46.32.7 with SMTP id g7ls824828ljg.14.gmail; Fri, 21 Jul 2017 00:31:06 -0700 (PDT) X-Received: by 10.25.39.211 with SMTP id n202mr900808lfn.16.1500622266541; Fri, 21 Jul 2017 00:31:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1500622266; cv=none; d=google.com; s=arc-20160816; b=jbp6gWUY3d+5K3kxpFbNRWVCvtIKHpf3sMila7I3FYbeX9ukJ+G7m+NgBvhsobDQco L6nEQHwbUCy0mklCfqprZAGWY3J6J53cBQWUQyIci6E8eGdEiFhpp55kK1gwBHskI8iy gkcKHw3RT2GQ582dC5mcPSERZ0knaJxlMJtBol18iJ9FSW4LL9I2PMJCOUzF9acK8njy mYq8rJHmXSgg8CTvm2y3jaR2vDVuUOM4hsUyLk9Z1iIVB+TzchgyXu0rUDTf8FmMEe0P f9Un+b23SUMZZigoFfRSgsj8u7azV2MRLC1yd1aW9sSiwxPKN4VqsSe/JsdHmTsbLxBU O6IQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :arc-authentication-results; bh=PnOO7sq4N4R9qASxgoXuij1AFarEm8O9DXWnltwok8g=; b=ObILBTfJ2pqlsM9rYXuwD2x+v3lLjnju0e2FAhuWxkqYTb5LyTJaQjdq2VnwMJXFuK inIOHCOECDRD6EGQ7AvGMgjtC/cVaWFIlgvHWZ9e9YDEKF3KxynQspyLtRux9Ng0ZV7e KAjFnnGkqiZ3/bDPEfd5nYKbQjIxIcVvKzh+RfNOyhk973wswGqyGlyiJJmqmiyMUH4q IDyBSiA1siLEwM94EZdLNCJoY7RBxf2utt0EWefRWIzmp1jj1h926k82Ep6JejS6EyIL iLp1WboQFNTbFAUrNFtXk0S1mCzWQVwuF3YfthJpvhgfH4CzFM/qwaTQmNeWUWEgo7q6 +ogA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 192.35.17.14 is neither permitted nor denied by best guess record for domain of jan.kiszka@siemens.com) smtp.mailfrom=jan.kiszka@siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id m85si128448wmi.7.2017.07.21.00.31.06 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Jul 2017 00:31:06 -0700 (PDT) Received-SPF: neutral (google.com: 192.35.17.14 is neither permitted nor denied by best guess record for domain of jan.kiszka@siemens.com) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 192.35.17.14 is neither permitted nor denied by best guess record for domain of jan.kiszka@siemens.com) smtp.mailfrom=jan.kiszka@siemens.com Received: from mail3.siemens.de (mail3.siemens.de [139.25.208.14]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id v6L7V6AC013743 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 21 Jul 2017 09:31:06 +0200 Received: from md1f2u6c.ww002.siemens.net ([139.22.121.131]) by mail3.siemens.de (8.15.2/8.15.2) with ESMTP id v6L7V5IN014383; Fri, 21 Jul 2017 09:31:05 +0200 Subject: Re: [RFC] can we just extend openembedded to get Isar 2.0 To: isar-users@googlegroups.com, Henning Schild , Alexander Smirnov , KOBAYASHI Yoshitake , Daniel Sangorrin , kazuhiro3.hayashi@toshiba.co.jp References: <20170720103037.794aab2e@md1em3qc> <20170720230529.GB5040@yssyq.radix50.net> From: Jan Kiszka Message-ID: <844d35a9-6ffd-9a84-b324-21acec937eaf@siemens.com> Date: Fri, 21 Jul 2017 09:31:05 +0200 User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.8.1.12) Gecko/20080226 SUSE/2.0.0.12-1.1 Thunderbird/2.0.0.12 Mnenhy/0.7.5.666 MIME-Version: 1.0 In-Reply-To: <20170720230529.GB5040@yssyq.radix50.net> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: anETHOROm4MV On 2017-07-21 01:05, Baurzhan Ismagulov wrote: > Hello, > > thanks for all the ideas. > > The short answer is, Isar has started from scratch as a rootfs generator from > existing binary distro + self-built packages. In plain German ;) , OE is not > necessary for this purpose. OE is a distro, and Isar is a builder. OE-core is also a distro, but that is indeed the uninteresting part. We are interested in all the stuff in meta/classes, lib, you-name-it. All what is still missing and needs to be copied, adjusted, and MAINTAINED separately in isar so far. This gap is still to large, and we need to define a strategy now how to close it sustainably. > > Of course, this is not that simple. You, and other people at various > conferences, ask about the tools that you know and want to use, and this is > exactly what we care about: Re-use mainstream tools and don't invent our own > wheel. It's not just tools, it's also their infrastructure. The tools are easy to copy, but the infrastructure has dependencies that need to be continuously adjusted. The question is now, can we bend the OE-core towards building a Debian distro in the Debian way with its pre-existing infrastructure, or do we really have to fork it? > > What we are doing now is exactly bringing the tools many people are familiar > with into Debian(-based system) development. Where the modified tools are > hosted (OE vs. Isar repos) is secondary; the primary issue is to make them > useful for both worlds -- one has to do that also if one inherits from OE. > Developing this same feature requires just the same effort compared to adapting > the OE tools for Debian directly in OE repos. Our plan is to harmonize the > tools for both use cases, submit the patches, and see what the OE community > thinks about that. > > So, to answer your original question, I don't exclude the possibility of using > Isar on top of OE if it would ever make sense. However, keeping the possibility > of using plain Isar is important for commercial product builders. Rationale: > The short answer above, plus a major disadvantage: OE is huge, and this luggage > is a problem for CM and integrators (I had to maintain OE in ClearCase, and I'm > glad I don't anymore). OE-core is also just 58K LOC - smaller than bitbake itself, thus Isar. (And ClearCase is dead, just not buried everywhere ;).) > > Regarding bitbake, I don't agree it's retrofitting. Bitbake is a great generic > tool, perfectly suitable for the task. In fact, we build other commercial > embedded systems with bitbake, without Isar or OE. Other tools like wic, > runqemu, etc. are more OE-centric (mostly due to sysroot), but I'm glad to see > that the required changes were quite manageable. > > The problem with Debian support in OE is that it is very undebian. Debian has > its strengths, and there are reasons why Debian developers strongly dislike OE > (see e.g. [1], and I've heard many such discussions). Deby / meta-debian > follows the OE way of building Debian packages, and currently considers moving > to the Debian way. The reason is, as soon as you oeize a package, you have to > maintain it. That is why Isar does things the Debian way and aims at avoiding > massive package modifications; OE doesn't really help here. The idea is not to move to the OE way of doing Debian, for sure. It's about adding the right way to the OE-core tools and libs that are useful or even mandatory for Isar instead of forking or reimplementing them now. > > A couple of comments below. > > On Thu, Jul 20, 2017 at 11:15:09AM +0200, Jan Kiszka wrote: >>> But a lot of things are still missing. For true customization one would >>> need a way to smuggle files into the rootfs, patch files in there, >>> execute scripts in there. Ideally with collision protection and special >>> protection for config-files etc. > > Customization is implemented in configscript.sh. Some more structure could be > introduced, but we didn't need it till now, as the changes beyond custom debs > were reasonable in size. The problem is that Isar wasn't applied on a larger scale yet. And it was only tested against the bitbake-unlike template pattern, instead of using layering consequently. We are doing both now, and the problems surface. All fixable, but the question is, where to do that best. > > >>> Claudius already suggested building custom .debs in Isar to solve that. > > Yes. we built them where the effort was justified. No, this is about making this feature available as library (class) so that you write three lines of recipe and have a customization deb package. That effort would be minimal for the user, and we only need to solve this once properly. If OE-core cannot provide that, we will have to implement it ourselves. > > >>> Another thing i already came across is Image conversion .raw >>> -> .qcow2/.vdi/.vmdk all that is missing in Isar. Same was true for the >>> wic support until recently. > > As introduced above, Isar follows the Debian way to reuse as much as possible. > That is why just basing on OE won't make the tools work for the Isar layer OOTB > -- it will require the very adaptations we do now. > Can you provide a concrete code example? What all prevents using upstream wic (including its image class) more as-is and forces us to fork it? > >>> People that are coming from Yocto and want to switch to Isar will never >>> get their recipes to work, because all the utils-classes are missing in >>> Isar and would need to be ported/pulled-in. > > This is what we ideally don't want to have. It is possible to build packages in > a basic OE way, but the preferred way is to debianize all packages. We want to > have stuff like clean, etc., but duplicating the whole OE functionality doesn't > make sense and won't work, since Isar and OE have different base systems and > toolchains. Again, concrete examples would help to underline this high-level statement. We will be asked these questions over and over again, so we need the arguments ready at hand. > > >>> Isar needs sudo, and someone has to integrate libpseudo. > > This is on TODO. > It's actually WIP by us now - it's urgent because we need CI support, and that's in conflict with the sudo workarounds. > >>> Isar still has several small issues with correct rebuilds after config >>> changes etc. > > These issues are inherited from bitbake and its default setting of doing so. > IIRC, there was a var to change that behavior, but I couldn't remember its > name, have to dig the code. > > >>> OE solved all those problems already > > AFAIK, not the rebuild one. > But, e.g., it has proper support for SRC_URI and fetching, proxies, mirrors, image types, and possibly more we need. Are all of these feature so tangled up with OE assumptions that stubbing and adjusting their dependencies would be worse than reimplementing them? > >>> but what it can not do is >>> fetch .debs from a mirror and bundle them together with multistrap. OE >>> most likely contains a lot of other usefull stuff that we might need >>> eventually. > > You can copy the recipe from Isar, and OE will be able to multistrap. The > problem is, you can't do anything useful with that afterwards due to the > different base system and compiler. OE != Debian, just like Ubuntu != Debian. > OK, that is a more concrete example to discuss: assumption of OE-core about the compiler and the build environment (when we are building, the exceptional case). Where do they bite us when reusing OE-core to generate, e.g., images? Or fetching sources? Or any other case? Yes, 90% of OE-core may be unused with Isar. But even 10% would still be an order of magnitude more lines of code than Isar has right now. That's the concern. If we have to port all that, or even just half of it over, we need strong reasons, ideally documented. > >>> Does it really make sense for us to reinvent all those OE-features in >>> Isar, or should we just got the other way around and put the Isar >>> features into OE? > > The truth is, both ways are the same due to the reasons above (except where the > modified tools are hosted, which doesn't matter), and we are already going that > way. Doing on top of OE doesn't make the tools work OOTB. But it brings the OE > distro luggage, which is not necessary for Isar use cases. > > >>> At a first glance it looks like you have to teach OE to get debian >>> packages and create a rootfs from them. Tell it to not compile anything >>> at first. Probably much like the do_rootfs from Isar. We might get away >>> with a layer on top of OE and maybe a few patches to OE that can maybe >>> be mainlined. Or maybe the whole thing could become mainline OE some >>> day. > > IIUC, not compiling while keeping the whole vars intact would make that a > mutant behaving against the expectations of the users. Changing the vars would > be creating a small Isar inside OE; I don't think this can be done in a layer > alone. Regarding mainlining -- we want to start with the tools. I doubt the two > paradigms (distro vs. builder) can / should be aligned into a single system. > > > 1. https://lists.debian.org/debian-embedded/2016/04/msg00013.html > > > With kind regards, > Baurzhan. > Thanks, Jan -- Siemens AG, Corporate Technology, CT RDA ITP SES-DE Corporate Competence Center Embedded Linux