Re: [RFC] Image templates

[Jan Kiszka <jan.kiszka@siemens.com>]

On 03.12.20 13:23, Claudius Heine wrote:
> Hi Joe,
> 
> On 2020-12-02 17:37, Joe MacDonald wrote:
>> Hi Claudius,
>>
>> [Re: [RFC] Image templates] On 20.12.02 (Wed 14:00) Claudius Heine wrote:
>>
>>> Hi Cedric,
>>>
>>> On 2020-12-01 08:59, Cedric Hombourger wrote:
>>>> Image Templates for ISAR
>>>> ========================
>>>>
>>>> Introduction
>>>> ------------
>>>>
>>>> Embedded System projects may have multiple teams working at different
>>>> levels. It is not uncommon to have groups creating a "base platform"
>>>> which is handed over to product teams to customize further.
>>>> Thanks to ISAR leveraging bitbake, image recipes from a base layer may
>>>> be easily extended by product layers but also with its ability to
>>>> produce and use a `base-apt` instead of upstream package feeds.
>>>> This workflow functions well in many cases, it may not be optimal for
>>>> application teams. Such teams are typically receiving a base platform
>>>> release that is likely to be left untouched for weeks.
>>>> Creating a product image for those teams is sometimes as easy as adding
>>>> a few packages.
>>>>
>>>> In such cases, it may be desirable to have a mechanism for ISAR to
>>>> construct product images starting from the base platform root
>>>> file-system instead of ISAR's bootstrap root file-system. We will refer
>>>> to these base platform root file-systems as _Template Images_.
>>>
>>> In principle I understand your requirement, however I don't think
>>> Isar (and
>>> possible Debian) is the right base for this.
>>>
>>> What you probably require is some sort of functional and non-destructive
>>> root-fs build approach to root file system, that lets you arbitrarily
>>> cache
>>> and distribute each stage of the root fs build. Generating and
>>> reusing the
>>> 'base-apt' repository is much easier than it would be for root file
>>> systems.
>>>
>>> Also stuff like docker solves only a small part of that solution.
>>>
>>> You will probably have to build a new system inspired by stuff like the
>>> bazel build system[1], and nixox[2] or guixos[3] and maybe even stuff
>>> like
>>> ostree [4].
>>
>> Thank you for the suggestions.  I had looked at bazel a while back for
>> something unrelated but my impression of it is really that it's aimed
>> squarely at application development teams, so never really investigated
>> how they do their builds.
> 
> The list is more for inspiration sake, not to actually use any of those
> technologies.
> 
> You are right Bazel is not really useful for this kind of work. I also
> don't have much experience with this, but AFAIK they are more of a
> Copy-on-Write build system. Which might be a good idea, because that
> would allow you to share partial build products.
> 
>>  We had a proof-of-concept working about a year
>> ago with ostree on top of Isar but ultimately that didn't seem to be a
>> good fit either since it placed significant restrictions on the
>> filesystem.  In the intervening time the project has also moved in a more
>> container-focused direction, so it also doesn't seem to address this
>> space
>> any longer.
> 
> Again my point was not using ostree directly, but to let you be inspired
> by it. What they did was putting a version of root file systems, that
> lets you reuse parts and create small deltas.
> 
>> NixOS and Guix both warrant further investigation, I think, though I've
>> done a bit of preliminary work with NixOS and I think the differences
>> between it and Debian in package format and maintenance tools are too
>> great to make it practical.
> 
> NixOS lets you download pre-build artifacts at the same time as allowing
> you to add custom patches to packages, causing it to just rebuild only
> what you changed, which would again be a pretty great attribut to have
> for such a system.
> 
>> In short, I definitely agree this is potentially a significant adjustment
>> and a large undertaking, but Isar does also seem almost uniquely
>> positioned to make this work, and why whe're making this proposal and
>> looking for feedback.
> 
> Isar is based on Debian, and while Debian is a great general
> Distribution, I don't think that pick and choosing different rootfs
> templates with different versions together would be possible. Which
> would IMO be a central property for such a system.
> 
> I understand that this proposal only considers a multiple images based
> on one template (1:m), because doing it otherwise could not really be
> solved here. But the modularity of bitbake based projects with OE and
> Isar is about m:n, e.g. multiple layers can be used and reused for
> multiple images.
> 
>>> The rest of what you describe sounds like they would try to shoehorn
>>> this
>>> into the existing isar build process and then trying to explicitly
>>> fix just
>>> some of the many future issues you will run into, instead of actually
>>> designing something that is actually able to deal with all those
>>> issues in a
>>> generic way.
>>
>> Are you thinking of something specific in the design that you see being
>> long-term problematic, or is this more of a general concern like "this is
>> plainly out of scope for Isar"?  Our objective is to produce something
>> that compliments and extends existing workflows, not to bolt on something
>> that doesn't make sense, so we're open to all feedback and views of what
>> could be danger areas or sources for future pain.
> 
> You will have to deal also with removal of packages that where installed
> in a template, and possible allow to patch and rebuild packages that are
> part of the template. That would mean the packages written in other all
> other isar layers now could cause a whole new dimension of issues. Isar
> packages don't really have the same QA as packages that are part of
> Debian and might not work well with removal. (This is also pointed out
> by Henning)
> 
> No idea how you would deal with detecting if some packages needs to be
> rebuild, because a patch was added. You will probably have to deal with
> signatures and sstate..
> 
>>
>>> Personally, I would be interested in such a new system, but that will
>>> take
>>> serious dedication to design and implement it properly.
>>>
>>> Changing that now in Isar will probably break a lot of assumptions and
>>> therefore limit its usefulness.
>>
>> That's definitley not our goal and the approach we are proposing, done as
>> a new class, intends to ensure it doesn't spider out and touch things
>> that
>> will impact unrelated Isar workflows.  If you can point to areas where
>> we're potentially breaking assumptions in Isar, most likely around image
>> assembly, but maybe other oareas, we'd be very happy to refine our view
>> and goals.
> 
> - Packages in Isar don't have the same QA as packages in Debian, and
> might not work with removal, because the assumption was that if they
> should not be part of an image, they would not have been installed.
> - Packages that are part of the template might need to be recompiled,
> because the image itself installed a different version of a library.
> This is solved in isar, because it would just build everything and let
> apt figure out the correct dependencies and installs all packages, but
> with this, you don't really have a way to call back into isar to compile
> some more packages. (e.g. kernel modules that are part of the template
> image, and I make a small, stable kernel update in the depending image)
> 
> I would expect a lot more of those...

I think a key element of any templating concept is defining its scope -
and limits. This cannot become a generic solution for all the scenarios
we currently model with a single image build. I also do not think it
will work for rebuilding random packages that have complex dependencies
- that's already hairy with plain Isar (think of rebuilding an essential
package...).

Probably the best and the cleanest we can do is to define that it is a
post-imaging package addition or removal step, acting fully within the
Debian packaging world. That will require additional QA on self-built
packages to make them cleanly deinstall as Claudius pointed out, but
that has a value of its own.

Template-based image post-processing may also be seen like
cross-compiling: known to work is native package compilation, so this
should be the production mode (though I know that many images use it for
production now). Analogously, the image template post-processing
approach should not be used to generate production images (famous last
words).

And that brings us to the technical requirements on generating a
template and building on top of it:

 - bundle all needed artifacts (template image AND buildchroots)
 - write a recipe that takes those artifacts and act as an alternative
   dependency to dpkg*-based recipes
 - write a recipe that takes packages, self-built or repo-provided, and
   installs them into a template, just like you would on the running
   device (alternative imaging target)
 - introduce some control mode for Isar to activate all this, switching
   dependencies accordingly

/Maybe/ we can also hook after rootfs generation but before imaging,
rerunning wic or other image classes also on the template. But that
would mandate even more that the post-processing will run over the very
same layer (sha or whatever signature-anchored) as the template was
generated by. But I would start as simple as possible.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux