From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6901194598360547328 X-Received: by 2002:a2e:8315:: with SMTP id a21mr7861311ljh.29.1607321895945; Sun, 06 Dec 2020 22:18:15 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:3614:: with SMTP id d20ls475080lja.6.gmail; Sun, 06 Dec 2020 22:18:14 -0800 (PST) X-Google-Smtp-Source: ABdhPJy/ysl/2sCBm5oWMrp26jGJYE+PW2OJDnmMOH1annxUPiBaGtToaOvLWnHTtRf3SoiE+JTg X-Received: by 2002:a2e:6c0f:: with SMTP id h15mr728048ljc.305.1607321894724; Sun, 06 Dec 2020 22:18:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1607321894; cv=none; d=google.com; s=arc-20160816; b=tl47J7l1arozRptrYJ+JOfNj131XZSxRxGl2HGkscW3bIZpTyqZBBjY/rEcgIhF6DA r00tHL+xEicj4yuIMxXwyKCLyde7XZHbUwkHnXVYuRKaFaxUkcBXVQkgWfEdvEcAcYO0 /TtMFOj8LXuyN3CWg9yDJDQW4VO4Y8OenWzNSGNJy4+1uCIjEujKU1YxCAAMsTmBbvI7 TPZqh+CbQpDqU/M0BBCbwvzaqJwIcOIphNzAPE8lkWGYXoU2FWNjOhovTn8RpfKPLc6f eNIOhm9nHOmxvSaDd7bmrpMTaoevX6np7PtBACKdff4QJrFLsJaAi7uL/Sn7MDF/7Eb7 TYrg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject; bh=7ArnJEeyCDUsrzDpDXbW4ZBVS5nrSjchCMK0iOR2dyo=; b=cI0YWG834FR7gigI7KcthL5/lmPX8STGHo34BTWYIi6YZ6es9MEJhqXN8Wz+qXMNQL F45uOpSZ+l+Yp4ytNE1NDpu70/7xLP3wv5FwRAPHFuLKcUqkMTQp0rQ5g00a349wvDVE Lt8tH6gjubDlvzvUazlQrQyG2N5g5v+nKpRKc9X8iL5743Qh/8QqyawUkP9Bs0XEeR83 BFIOvKFFoUtkrLb0uMmrYDqqzEKx9D6qRHpdbfHdlymJAhkyjzKcIM0Kg9Hk/Vk6ykEK m3qk5lAyifdaDKM6eXkosgvFvkMb1xdoN7GJ/DgqzIO5FAH1xiR2lNsDjYotMgCotxmu KjGg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.40 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from gecko.sbs.de (gecko.sbs.de. [194.138.37.40]) by gmr-mx.google.com with ESMTPS id o13si491746lfo.5.2020.12.06.22.18.14 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 06 Dec 2020 22:18:14 -0800 (PST) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.40 as permitted sender) client-ip=194.138.37.40; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.40 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by gecko.sbs.de (8.15.2/8.15.2) with ESMTPS id 0B76ICLH029017 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 7 Dec 2020 07:18:12 +0100 Received: from [167.87.36.107] ([167.87.36.107]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id 0B76IBSw025917; Mon, 7 Dec 2020 07:18:11 +0100 Subject: Re: [RFC] Image templates To: Claudius Heine , Joe MacDonald Cc: Cedric Hombourger , isar-users@googlegroups.com References: <6bf9a786-51b8-2430-90f5-2857746d003f@mentor.com> <20201202163731.GC7804@mentor.com> <3ad50365-6298-8ba3-f2ce-39f5f6210fe9@denx.de> From: Jan Kiszka Message-ID: Date: Mon, 7 Dec 2020 07:18:11 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.5.0 MIME-Version: 1.0 In-Reply-To: <3ad50365-6298-8ba3-f2ce-39f5f6210fe9@denx.de> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: hM6OU7Ori9Zw On 03.12.20 13:23, Claudius Heine wrote: > Hi Joe, > > On 2020-12-02 17:37, Joe MacDonald wrote: >> Hi Claudius, >> >> [Re: [RFC] Image templates] On 20.12.02 (Wed 14:00) Claudius Heine wrote: >> >>> Hi Cedric, >>> >>> On 2020-12-01 08:59, Cedric Hombourger wrote: >>>> Image Templates for ISAR >>>> ======================== >>>> >>>> Introduction >>>> ------------ >>>> >>>> Embedded System projects may have multiple teams working at different >>>> levels. It is not uncommon to have groups creating a "base platform" >>>> which is handed over to product teams to customize further. >>>> Thanks to ISAR leveraging bitbake, image recipes from a base layer may >>>> be easily extended by product layers but also with its ability to >>>> produce and use a `base-apt` instead of upstream package feeds. >>>> This workflow functions well in many cases, it may not be optimal for >>>> application teams. Such teams are typically receiving a base platform >>>> release that is likely to be left untouched for weeks. >>>> Creating a product image for those teams is sometimes as easy as adding >>>> a few packages. >>>> >>>> In such cases, it may be desirable to have a mechanism for ISAR to >>>> construct product images starting from the base platform root >>>> file-system instead of ISAR's bootstrap root file-system. We will refer >>>> to these base platform root file-systems as _Template Images_. >>> >>> In principle I understand your requirement, however I don't think >>> Isar (and >>> possible Debian) is the right base for this. >>> >>> What you probably require is some sort of functional and non-destructive >>> root-fs build approach to root file system, that lets you arbitrarily >>> cache >>> and distribute each stage of the root fs build. Generating and >>> reusing the >>> 'base-apt' repository is much easier than it would be for root file >>> systems. >>> >>> Also stuff like docker solves only a small part of that solution. >>> >>> You will probably have to build a new system inspired by stuff like the >>> bazel build system[1], and nixox[2] or guixos[3] and maybe even stuff >>> like >>> ostree [4]. >> >> Thank you for the suggestions.  I had looked at bazel a while back for >> something unrelated but my impression of it is really that it's aimed >> squarely at application development teams, so never really investigated >> how they do their builds. > > The list is more for inspiration sake, not to actually use any of those > technologies. > > You are right Bazel is not really useful for this kind of work. I also > don't have much experience with this, but AFAIK they are more of a > Copy-on-Write build system. Which might be a good idea, because that > would allow you to share partial build products. > >>  We had a proof-of-concept working about a year >> ago with ostree on top of Isar but ultimately that didn't seem to be a >> good fit either since it placed significant restrictions on the >> filesystem.  In the intervening time the project has also moved in a more >> container-focused direction, so it also doesn't seem to address this >> space >> any longer. > > Again my point was not using ostree directly, but to let you be inspired > by it. What they did was putting a version of root file systems, that > lets you reuse parts and create small deltas. > >> NixOS and Guix both warrant further investigation, I think, though I've >> done a bit of preliminary work with NixOS and I think the differences >> between it and Debian in package format and maintenance tools are too >> great to make it practical. > > NixOS lets you download pre-build artifacts at the same time as allowing > you to add custom patches to packages, causing it to just rebuild only > what you changed, which would again be a pretty great attribut to have > for such a system. > >> In short, I definitely agree this is potentially a significant adjustment >> and a large undertaking, but Isar does also seem almost uniquely >> positioned to make this work, and why whe're making this proposal and >> looking for feedback. > > Isar is based on Debian, and while Debian is a great general > Distribution, I don't think that pick and choosing different rootfs > templates with different versions together would be possible. Which > would IMO be a central property for such a system. > > I understand that this proposal only considers a multiple images based > on one template (1:m), because doing it otherwise could not really be > solved here. But the modularity of bitbake based projects with OE and > Isar is about m:n, e.g. multiple layers can be used and reused for > multiple images. > >>> The rest of what you describe sounds like they would try to shoehorn >>> this >>> into the existing isar build process and then trying to explicitly >>> fix just >>> some of the many future issues you will run into, instead of actually >>> designing something that is actually able to deal with all those >>> issues in a >>> generic way. >> >> Are you thinking of something specific in the design that you see being >> long-term problematic, or is this more of a general concern like "this is >> plainly out of scope for Isar"?  Our objective is to produce something >> that compliments and extends existing workflows, not to bolt on something >> that doesn't make sense, so we're open to all feedback and views of what >> could be danger areas or sources for future pain. > > You will have to deal also with removal of packages that where installed > in a template, and possible allow to patch and rebuild packages that are > part of the template. That would mean the packages written in other all > other isar layers now could cause a whole new dimension of issues. Isar > packages don't really have the same QA as packages that are part of > Debian and might not work well with removal. (This is also pointed out > by Henning) > > No idea how you would deal with detecting if some packages needs to be > rebuild, because a patch was added. You will probably have to deal with > signatures and sstate.. > >> >>> Personally, I would be interested in such a new system, but that will >>> take >>> serious dedication to design and implement it properly. >>> >>> Changing that now in Isar will probably break a lot of assumptions and >>> therefore limit its usefulness. >> >> That's definitley not our goal and the approach we are proposing, done as >> a new class, intends to ensure it doesn't spider out and touch things >> that >> will impact unrelated Isar workflows.  If you can point to areas where >> we're potentially breaking assumptions in Isar, most likely around image >> assembly, but maybe other oareas, we'd be very happy to refine our view >> and goals. > > - Packages in Isar don't have the same QA as packages in Debian, and > might not work with removal, because the assumption was that if they > should not be part of an image, they would not have been installed. > - Packages that are part of the template might need to be recompiled, > because the image itself installed a different version of a library. > This is solved in isar, because it would just build everything and let > apt figure out the correct dependencies and installs all packages, but > with this, you don't really have a way to call back into isar to compile > some more packages. (e.g. kernel modules that are part of the template > image, and I make a small, stable kernel update in the depending image) > > I would expect a lot more of those... I think a key element of any templating concept is defining its scope - and limits. This cannot become a generic solution for all the scenarios we currently model with a single image build. I also do not think it will work for rebuilding random packages that have complex dependencies - that's already hairy with plain Isar (think of rebuilding an essential package...). Probably the best and the cleanest we can do is to define that it is a post-imaging package addition or removal step, acting fully within the Debian packaging world. That will require additional QA on self-built packages to make them cleanly deinstall as Claudius pointed out, but that has a value of its own. Template-based image post-processing may also be seen like cross-compiling: known to work is native package compilation, so this should be the production mode (though I know that many images use it for production now). Analogously, the image template post-processing approach should not be used to generate production images (famous last words). And that brings us to the technical requirements on generating a template and building on top of it: - bundle all needed artifacts (template image AND buildchroots) - write a recipe that takes those artifacts and act as an alternative dependency to dpkg*-based recipes - write a recipe that takes packages, self-built or repo-provided, and installs them into a template, just like you would on the running device (alternative imaging target) - introduce some control mode for Isar to activate all this, switching dependencies accordingly /Maybe/ we can also hook after rootfs generation but before imaging, rerunning wic or other image classes also on the template. But that would mandate even more that the post-processing will run over the very same layer (sha or whatever signature-anchored) as the template was generated by. But I would start as simple as possible. Jan -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux