Re: [RFC] Image templates

[Claudius Heine <ch@denx.de>]

Hi Joe,

On 2020-12-02 17:37, Joe MacDonald wrote:
> Hi Claudius,
> 
> [Re: [RFC] Image templates] On 20.12.02 (Wed 14:00) Claudius Heine wrote:
> 
>> Hi Cedric,
>>
>> On 2020-12-01 08:59, Cedric Hombourger wrote:
>>> Image Templates for ISAR
>>> ========================
>>>
>>> Introduction
>>> ------------
>>>
>>> Embedded System projects may have multiple teams working at different
>>> levels. It is not uncommon to have groups creating a "base platform"
>>> which is handed over to product teams to customize further.
>>> Thanks to ISAR leveraging bitbake, image recipes from a base layer may
>>> be easily extended by product layers but also with its ability to
>>> produce and use a `base-apt` instead of upstream package feeds.
>>> This workflow functions well in many cases, it may not be optimal for
>>> application teams. Such teams are typically receiving a base platform
>>> release that is likely to be left untouched for weeks.
>>> Creating a product image for those teams is sometimes as easy as adding
>>> a few packages.
>>>
>>> In such cases, it may be desirable to have a mechanism for ISAR to
>>> construct product images starting from the base platform root
>>> file-system instead of ISAR's bootstrap root file-system. We will refer
>>> to these base platform root file-systems as _Template Images_.
>>
>> In principle I understand your requirement, however I don't think Isar (and
>> possible Debian) is the right base for this.
>>
>> What you probably require is some sort of functional and non-destructive
>> root-fs build approach to root file system, that lets you arbitrarily cache
>> and distribute each stage of the root fs build. Generating and reusing the
>> 'base-apt' repository is much easier than it would be for root file systems.
>>
>> Also stuff like docker solves only a small part of that solution.
>>
>> You will probably have to build a new system inspired by stuff like the
>> bazel build system[1], and nixox[2] or guixos[3] and maybe even stuff like
>> ostree [4].
> 
> Thank you for the suggestions.  I had looked at bazel a while back for
> something unrelated but my impression of it is really that it's aimed
> squarely at application development teams, so never really investigated
> how they do their builds.

The list is more for inspiration sake, not to actually use any of those 
technologies.

You are right Bazel is not really useful for this kind of work. I also 
don't have much experience with this, but AFAIK they are more of a 
Copy-on-Write build system. Which might be a good idea, because that 
would allow you to share partial build products.

>  We had a proof-of-concept working about a year
> ago with ostree on top of Isar but ultimately that didn't seem to be a
> good fit either since it placed significant restrictions on the
> filesystem.  In the intervening time the project has also moved in a more
> container-focused direction, so it also doesn't seem to address this space
> any longer.

Again my point was not using ostree directly, but to let you be inspired 
by it. What they did was putting a version of root file systems, that 
lets you reuse parts and create small deltas.

> NixOS and Guix both warrant further investigation, I think, though I've
> done a bit of preliminary work with NixOS and I think the differences
> between it and Debian in package format and maintenance tools are too
> great to make it practical.

NixOS lets you download pre-build artifacts at the same time as allowing 
you to add custom patches to packages, causing it to just rebuild only 
what you changed, which would again be a pretty great attribut to have 
for such a system.

> In short, I definitely agree this is potentially a significant adjustment
> and a large undertaking, but Isar does also seem almost uniquely
> positioned to make this work, and why whe're making this proposal and
> looking for feedback.

Isar is based on Debian, and while Debian is a great general 
Distribution, I don't think that pick and choosing different rootfs 
templates with different versions together would be possible. Which 
would IMO be a central property for such a system.

I understand that this proposal only considers a multiple images based 
on one template (1:m), because doing it otherwise could not really be 
solved here. But the modularity of bitbake based projects with OE and 
Isar is about m:n, e.g. multiple layers can be used and reused for 
multiple images.

>> The rest of what you describe sounds like they would try to shoehorn this
>> into the existing isar build process and then trying to explicitly fix just
>> some of the many future issues you will run into, instead of actually
>> designing something that is actually able to deal with all those issues in a
>> generic way.
> 
> Are you thinking of something specific in the design that you see being
> long-term problematic, or is this more of a general concern like "this is
> plainly out of scope for Isar"?  Our objective is to produce something
> that compliments and extends existing workflows, not to bolt on something
> that doesn't make sense, so we're open to all feedback and views of what
> could be danger areas or sources for future pain.

You will have to deal also with removal of packages that where installed 
in a template, and possible allow to patch and rebuild packages that are 
part of the template. That would mean the packages written in other all 
other isar layers now could cause a whole new dimension of issues. Isar 
packages don't really have the same QA as packages that are part of 
Debian and might not work well with removal. (This is also pointed out 
by Henning)

No idea how you would deal with detecting if some packages needs to be 
rebuild, because a patch was added. You will probably have to deal with 
signatures and sstate..

> 
>> Personally, I would be interested in such a new system, but that will take
>> serious dedication to design and implement it properly.
>>
>> Changing that now in Isar will probably break a lot of assumptions and
>> therefore limit its usefulness.
> 
> That's definitley not our goal and the approach we are proposing, done as
> a new class, intends to ensure it doesn't spider out and touch things that
> will impact unrelated Isar workflows.  If you can point to areas where
> we're potentially breaking assumptions in Isar, most likely around image
> assembly, but maybe other oareas, we'd be very happy to refine our view
> and goals.

- Packages in Isar don't have the same QA as packages in Debian, and 
might not work with removal, because the assumption was that if they 
should not be part of an image, they would not have been installed.
- Packages that are part of the template might need to be recompiled, 
because the image itself installed a different version of a library. 
This is solved in isar, because it would just build everything and let 
apt figure out the correct dependencies and installs all packages, but 
with this, you don't really have a way to call back into isar to compile 
some more packages. (e.g. kernel modules that are part of the template 
image, and I make a small, stable kernel update in the depending image)

I would expect a lot more of those...

regards,
Claudius