[RFC] Image templates

[RFC] Image templates

[Cedric Hombourger]

Image Templates for ISAR
========================

Introduction
------------

Embedded System projects may have multiple teams working at different levels. It is not uncommon to have groups creating a "base platform" which is handed over to product teams to customize further.
Thanks to ISAR leveraging bitbake, image recipes from a base layer may be easily extended by product layers but also with its ability to produce and use a `base-apt` instead of upstream package feeds.
This workflow functions well in many cases, it may not be optimal for application teams. Such teams are typically receiving a base platform release that is likely to be left untouched for weeks.
Creating a product image for those teams is sometimes as easy as adding a few packages.

In such cases, it may be desirable to have a mechanism for ISAR to construct product images starting from the base platform root file-system instead of ISAR's bootstrap root file-system. We will refer
to these base platform root file-systems as _Template Images_.

This document will describe both the construction and the use of _template images_.

Construction
------------

Developers needing to produce image templates in addition to bootable system images will have their image recipes `inherit` the `produce-image-template` class either as part of their image development
or by extending an existing image with an appropriate `.bbappend`.

The class would invoke a `do_rootfs_template` task after `do_rootfs_install` and before the post-processing tasks (which should be applied on the final image). This new task would simply create a
template image in the deploy folder built from the common root file-system artifacts. Some systems may have additional, more extensive post-processing operations that must be performed on the final
image, this class will still allow for such operations to function normally.

It is anticipated that configuration files in `/etc/apt/sources.list.d` and `/etc/resolv.conf` should be excluded (they should be recreated while the template image is imported). A subsequent update
will define a criteria for determining which files should be excluded from the tarball. As yet the specific mechanism has not been determined, whether it would be a file list, pattern or some other
external attributes. We anticipate other objects that may need to be excluded are `/etc/hosts` and proxy configuration, to name two simple examples.

The process of constructing the base image would then continue as usual (with post-processing and imager tasks).

It is suggested to expose the produced image template as an opaque binary blob and use a `.tmpl.img` file extension although the initial implementation is anticipated to be a compressed tarball.
The template filesystem may need embedded meta-data to be consumed by import mechanism or image recipes that make use of templates but omitted from the final rootfs. It would be useful at the outset
to either specify a minimal set of metadata (build date, image name and feature list is the initial list to be considered) and the structure of that metadata (currently considering either JSON-formatted
suitable for extraction with `jq` or `deb` packaging, which has embedded metadata in the cpio archive).

Usage
-----

Application teams may create image recipes referencing image templates in their SRC_URI list. As noted above, image templates would use the `.tmpl.img` file extension. This would be used by ISAR's `rootfs`
class to locate image templates in the list and make sure that only one template is specified (extracting a template over another would have unpredictable results).

If the bulk of the template intelligence is present in the `do_rootfs` stages, the `do_rootfs_prepare` task from the `rootfs` class would behave differently.  In this case, an image template may be added to
an image SRC_URI. Instead of copying ISAR's bootstrap rootfs, the template feature would instead unpack the specified template. It would also be necessary to copy `/etc/apt/sources.list.d` from the bootstrap
and possibly other files (e.g. `/etc/resolv.conf`). As the environment may include different apt sources, a complete `apt-get update` would be in order.

ISAR would then "enter" the unpacked template carrying updated apt sources to install packages listed in `IMAGE_INSTALL` and `IMAGE_PREINSTALL` using `apt- get install` as it does today in `do_rootfs_install`.

This design should allow images constructed from an image template to produce a new image template (e.g. a base platform could provide a template having a BSP and a minimal Debian system, and middleware teams
could then produce their own template with middleware stacks added and finally consumed by application teams to produce a final system image).

Future Work
-----------

It may be desirable to offer a mechanism to remove packages coming from an image template (application teams may need to optimize the final system image to meet the footprint constraints of their design and/or
to reduce the attack surface by removing services they do not need).

Status
------

This high-level design was created to seek early feedback from the ISAR developer community by exposing the problem we are trying to solve and the mechanism we are proposing as a solution. The next step would be
the development of a Proof-of-Concept in the event where positive comments are received. This document may also be updated as feedback is received and may result in different approaches being discussed.

We will be happy to answer questions you may have or provide more details in areas where this document would be too evasive.

Re: [RFC] Image templates

[Jan Kiszka]

On 01.12.20 08:59, Cedric Hombourger wrote:
> Image Templates for ISAR
> ========================
> 
> Introduction
> ------------
> 
> Embedded System projects may have multiple teams working at different
> levels. It is not uncommon to have groups creating a "base platform"
> which is handed over to product teams to customize further.
> Thanks to ISAR leveraging bitbake, image recipes from a base layer may
> be easily extended by product layers but also with its ability to
> produce and use a `base-apt` instead of upstream package feeds.
> This workflow functions well in many cases, it may not be optimal for
> application teams. Such teams are typically receiving a base platform
> release that is likely to be left untouched for weeks.
> Creating a product image for those teams is sometimes as easy as adding
> a few packages.

There is another use case, even for base image developers: accelerating
the turn-around time when working only on few, widely isolated packages,
may it be the kernel or a bootloader artifact.

For that scenario, it should be possible to define an image template
excluding those bits, only injecting them on top in a second step.

> 
> In such cases, it may be desirable to have a mechanism for ISAR to
> construct product images starting from the base platform root
> file-system instead of ISAR's bootstrap root file-system. We will refer
> to these base platform root file-systems as _Template Images_.
> 
> This document will describe both the construction and the use of
> _template images_.
> 
> Construction
> ------------
> 
> Developers needing to produce image templates in addition to bootable
> system images will have their image recipes `inherit` the
> `produce-image-template` class either as part of their image development
> or by extending an existing image with an appropriate `.bbappend`.
> 
> The class would invoke a `do_rootfs_template` task after
> `do_rootfs_install` and before the post-processing tasks (which should
> be applied on the final image). This new task would simply create a
> template image in the deploy folder built from the common root
> file-system artifacts. Some systems may have additional, more extensive
> post-processing operations that must be performed on the final
> image, this class will still allow for such operations to function
> normally.
> 
> It is anticipated that configuration files in `/etc/apt/sources.list.d`
> and `/etc/resolv.conf` should be excluded (they should be recreated
> while the template image is imported). A subsequent update
> will define a criteria for determining which files should be excluded
> from the tarball. As yet the specific mechanism has not been determined,
> whether it would be a file list, pattern or some other
> external attributes. We anticipate other objects that may need to be
> excluded are `/etc/hosts` and proxy configuration, to name two simple
> examples.
> 
> The process of constructing the base image would then continue as usual
> (with post-processing and imager tasks).
> 
> It is suggested to expose the produced image template as an opaque
> binary blob and use a `.tmpl.img` file extension although the initial
> implementation is anticipated to be a compressed tarball.
> The template filesystem may need embedded meta-data to be consumed by
> import mechanism or image recipes that make use of templates but omitted
> from the final rootfs. It would be useful at the outset
> to either specify a minimal set of metadata (build date, image name and
> feature list is the initial list to be considered) and the structure of
> that metadata (currently considering either JSON-formatted
> suitable for extraction with `jq` or `deb` packaging, which has embedded
> metadata in the cpio archive).
> 
> Usage
> -----
> 
> Application teams may create image recipes referencing image templates
> in their SRC_URI list. As noted above, image templates would use the
> `.tmpl.img` file extension. This would be used by ISAR's `rootfs`
> class to locate image templates in the list and make sure that only one
> template is specified (extracting a template over another would have
> unpredictable results).
> 
> If the bulk of the template intelligence is present in the `do_rootfs`
> stages, the `do_rootfs_prepare` task from the `rootfs` class would
> behave differently.  In this case, an image template may be added to
> an image SRC_URI. Instead of copying ISAR's bootstrap rootfs, the
> template feature would instead unpack the specified template. It would
> also be necessary to copy `/etc/apt/sources.list.d` from the bootstrap
> and possibly other files (e.g. `/etc/resolv.conf`). As the environment
> may include different apt sources, a complete `apt-get update` would be
> in order.
> 
> ISAR would then "enter" the unpacked template carrying updated apt
> sources to install packages listed in `IMAGE_INSTALL` and
> `IMAGE_PREINSTALL` using `apt- get install` as it does today in
> `do_rootfs_install`.
> 
> This design should allow images constructed from an image template to
> produce a new image template (e.g. a base platform could provide a
> template having a BSP and a minimal Debian system, and middleware teams
> could then produce their own template with middleware stacks added and
> finally consumed by application teams to produce a final system image).
> 

There will likely be a number of traps we can stumble over, but the
general vision is shared.

One of the trap is that you may not only need a target image template
but also the corresponding buildchroots, if not isar-bootstraps.

Would your model be that a developer works within the layer that creates
the template? A key issue would be that changes to the layer that
invalidate the template would be detected and make the template-based
build fail or fall back to the full build.

Or would your model be that templating enforce a split build via
separate repos? That might be problematic from the maintenance
perspective as application injection will come before imaging, and the
latter is not the task of an application developer.

> Future Work
> -----------
> 
> It may be desirable to offer a mechanism to remove packages coming from
> an image template (application teams may need to optimize the final
> system image to meet the footprint constraints of their design and/or
> to reduce the attack surface by removing services they do not need).
> 
> Status
> ------
> 
> This high-level design was created to seek early feedback from the ISAR
> developer community by exposing the problem we are trying to solve and
> the mechanism we are proposing as a solution. The next step would be
> the development of a Proof-of-Concept in the event where positive
> comments are received. This document may also be updated as feedback is
> received and may result in different approaches being discussed.
> 
> We will be happy to answer questions you may have or provide more
> details in areas where this document would be too evasive.
> 

The whole thing also reminds me a lot of how docker images are created.
I wonder to which degree we can learn from that, reuse concepts or avoid
pitfalls. Adding Silvano with whom I was recently discussing base
container image generation via Isar.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [RFC] Image templates

[Henning Schild]

Did not even read it ... strong reject!

This smells like a mechanism to keep funny layers non OSS. While
allowing others to extend them ... without having their sources.
Leading to workaround hacks ... and issues in those "secret" layers not
being solved at their core.
I would first remove all "mel-*" packages before adding my own ;) ...
but i would be stuck with what postinst did break ... because they
likely dont prerm ...

Maybe i do not get it, you are jumping from a pretty "lame" motivation
right into implementation.

Let us look at the problems we currently have, who has them, why they
have them ... and later see what we could do.

I seriously doubt the idea that a "product" can be built by adding a
few packages to a base image. A prototype or dev image is more likely
...
And imagine yourself bumping a layer onto a new base-binary ... say
after a buster-bullseye, NetworkManager->systemd-networkd,
cron->systemd-timers jump ... good luck writing/reading that Changelog

Henning


Am Tue, 1 Dec 2020 08:59:20 +0100
schrieb Cedric Hombourger <Cedric_Hombourger@mentor.com>:

> Image Templates for ISAR
> ========================
> 
> Introduction
> ------------
> 
> Embedded System projects may have multiple teams working at different
> levels. It is not uncommon to have groups creating a "base platform"
> which is handed over to product teams to customize further. Thanks to
> ISAR leveraging bitbake, image recipes from a base layer may be
> easily extended by product layers but also with its ability to
> produce and use a `base-apt` instead of upstream package feeds. This
> workflow functions well in many cases, it may not be optimal for
> application teams. Such teams are typically receiving a base platform
> release that is likely to be left untouched for weeks. Creating a
> product image for those teams is sometimes as easy as adding a few
> packages.
> 
> In such cases, it may be desirable to have a mechanism for ISAR to
> construct product images starting from the base platform root
> file-system instead of ISAR's bootstrap root file-system. We will
> refer to these base platform root file-systems as _Template Images_.
> 
> This document will describe both the construction and the use of
> _template images_.
> 
> Construction
> ------------
> 
> Developers needing to produce image templates in addition to bootable
> system images will have their image recipes `inherit` the
> `produce-image-template` class either as part of their image
> development or by extending an existing image with an appropriate
> `.bbappend`.
> 
> The class would invoke a `do_rootfs_template` task after
> `do_rootfs_install` and before the post-processing tasks (which
> should be applied on the final image). This new task would simply
> create a template image in the deploy folder built from the common
> root file-system artifacts. Some systems may have additional, more
> extensive post-processing operations that must be performed on the
> final image, this class will still allow for such operations to
> function normally.
> 
> It is anticipated that configuration files in
> `/etc/apt/sources.list.d` and `/etc/resolv.conf` should be excluded
> (they should be recreated while the template image is imported). A
> subsequent update will define a criteria for determining which files
> should be excluded from the tarball. As yet the specific mechanism
> has not been determined, whether it would be a file list, pattern or
> some other external attributes. We anticipate other objects that may
> need to be excluded are `/etc/hosts` and proxy configuration, to name
> two simple examples.
> 
> The process of constructing the base image would then continue as
> usual (with post-processing and imager tasks).
> 
> It is suggested to expose the produced image template as an opaque
> binary blob and use a `.tmpl.img` file extension although the initial
> implementation is anticipated to be a compressed tarball. The
> template filesystem may need embedded meta-data to be consumed by
> import mechanism or image recipes that make use of templates but
> omitted from the final rootfs. It would be useful at the outset to
> either specify a minimal set of metadata (build date, image name and
> feature list is the initial list to be considered) and the structure
> of that metadata (currently considering either JSON-formatted
> suitable for extraction with `jq` or `deb` packaging, which has
> embedded metadata in the cpio archive).
> 
> Usage
> -----
> 
> Application teams may create image recipes referencing image
> templates in their SRC_URI list. As noted above, image templates
> would use the `.tmpl.img` file extension. This would be used by
> ISAR's `rootfs` class to locate image templates in the list and make
> sure that only one template is specified (extracting a template over
> another would have unpredictable results).
> 
> If the bulk of the template intelligence is present in the
> `do_rootfs` stages, the `do_rootfs_prepare` task from the `rootfs`
> class would behave differently.  In this case, an image template may
> be added to an image SRC_URI. Instead of copying ISAR's bootstrap
> rootfs, the template feature would instead unpack the specified
> template. It would also be necessary to copy
> `/etc/apt/sources.list.d` from the bootstrap and possibly other files
> (e.g. `/etc/resolv.conf`). As the environment may include different
> apt sources, a complete `apt-get update` would be in order.
> 
> ISAR would then "enter" the unpacked template carrying updated apt
> sources to install packages listed in `IMAGE_INSTALL` and
> `IMAGE_PREINSTALL` using `apt- get install` as it does today in
> `do_rootfs_install`.
> 
> This design should allow images constructed from an image template to
> produce a new image template (e.g. a base platform could provide a
> template having a BSP and a minimal Debian system, and middleware
> teams could then produce their own template with middleware stacks
> added and finally consumed by application teams to produce a final
> system image).
> 
> Future Work
> -----------
> 
> It may be desirable to offer a mechanism to remove packages coming
> from an image template (application teams may need to optimize the
> final system image to meet the footprint constraints of their design
> and/or to reduce the attack surface by removing services they do not
> need).
> 
> Status
> ------
> 
> This high-level design was created to seek early feedback from the
> ISAR developer community by exposing the problem we are trying to
> solve and the mechanism we are proposing as a solution. The next step
> would be the development of a Proof-of-Concept in the event where
> positive comments are received. This document may also be updated as
> feedback is received and may result in different approaches being
> discussed.
> 
> We will be happy to answer questions you may have or provide more
> details in areas where this document would be too evasive.
> 

Re: [RFC] Image templates

[Cedric Hombourger]

Hi Henning

On 12/1/20 10:13 AM, Henning Schild wrote:
> Did not even read it ... strong reject!
Well you should :) We are not saying that our vision is 100% right or 
even complete but when some of users came to us to request something 
like this, we felt that it definitely made sense. We viewed this as a 
docker-type development workflow. Maybe we can make the problem 
statement more clear to make sure we all understand the what before we 
get into the how.
> This smells like a mechanism to keep funny layers non OSS. While
> allowing others to extend them ... without having their sources.
Not at all. People shipping a base platform would evidently be aware of 
obligations they have to fulfill because of GPL and the like. Having a 
mechanism where we could tell the build to either get the prebuilt image 
template (sstate?) or build it from sources so that everything follows 
the workflow we have today is definitely something we would want (I think).
> Leading to workaround hacks ... and issues in those "secret" layers not
> being solved at their core.
Let's not jump into conclusions too quickly please. This is just a very 
high-level *vision* document submitted as a RFC (which maybe should 
stand for Read Fully and Comment :)). We can certainly discuss the 
use-case (which did not come from either Joe or I by the way), get some 
level of agreement that it is something we'd like to support and for 
sure we can discuss the how. I am sure there are different ways of 
caching mechanism for "base" images could be provided (to speed-up 
builds) and the outline we provided may not be the route this project 
may want to take. It is certainly fine if people here conclude that ISAR 
would not support this use-case. I would like the discussion to remain 
constructive and please do not suggest that we are trying to abuse the 
mechanism that we are suggesting here because that's definitely not the 
intent, never has and never will be.
> I would first remove all "mel-*" packages before adding my own ;) ...
> but i would be stuck with what postinst did break ... because they
> likely dont prerm ...
some of users are happy with the "base" they are getting and do not see 
a value in rebuilding it because it consumes significant time and as Jan 
suggested, some people just want to build a new image with their package 
version N+1 and don't really need to reassemble everything underneath 
each time. But yes, there should a mechanism to remove packages from the 
base if you wish, we certainly agree here. docker also provides a 
similar mechanism. I am sure we will find packages that are not 
friendly. The example of a missing prerm is a good one. Let's be clear, 
packages that I have seen created with Isar by several people/companies 
mostly focus on the do_rootfs_install / do_rootfs_postprocess flow. Many 
of those packages wouldn't play well in a pristine Debian environment. I 
would view this as an opportunity to get those packages to be more 
Debian compliant than they are today. In fact, I am very routinely 
recommending people I am working with to avoid using Isar for the 
purpose of creating a Debian. In my very humble opinion, packages should 
be designed/implemented/tested first in a pristine Debian environment 
and integrated into your Isar workflow (via an "inherit dpkg" recipe) 
when they are ready (while the variable injection mechanism provided by 
ISAR/bitbake is very powerful, it is also a recipe for creating 
non-reproducible packages or packages that aren't buildable without 
ISAR). Anyhow, this would be an entire new topic (I think); I would 
recommend for the sake of this discussion that we assume that packages 
are all "compliant" / "well-written"
> Maybe i do not get it, you are jumping from a pretty "lame" motivation
> right into implementation.
Not a singe line of code was written. we wanted to socialize the idea 
first. If we eventually get into an agreement of the use-case and 
hopefully vision of how this would look like at 30,000 feet, the next 
step would be the creation of a detailed design (possibly with some PoC 
code to validate the plan).
>
> Let us look at the problems we currently have, who has them, why they
> have them ... and later see what we could do.

I agree wholeheartedly.

Problem statement as received/understood from our user-base: "our system 
image is quite large because it includes a full desktop, an office 
suite, a browser and many other things" (user/customer did ask for all 
of them to be included). "Builds are taking a lot of time, we would like 
to have the build system just chroot into the existing image and install 
our new packages right away instead of recreating the rootfs from 
scratch because we are not touching the base anyway"

So maybe image templates isn't the way to go and some mechanism that 
would (1) update the existing package database in the rootfs, (2) use 
apt to upgrade packages that are already in there and (3) install any 
additional packages could be the way to go.

We are happy to throw away our initial proposal and create a new one if 
you prefer to stay away from the docker-model and instead focus on e.g. 
an incremental rootfs build or any other way being proposed here to 
speed up the build when just a few packages are either modified or added 
(and maybe removed as well?)

>
> I seriously doubt the idea that a "product" can be built by adding a
> few packages to a base image. A prototype or dev image is more likely
well the docker model has been quite successful ? :) but again, if we 
agree that an incremental rootfs build is the way to go (or any other) 
then you will definitely get our support and we are hoping that we could 
contribute. Our primary objective to find a solution to the problem that 
was reported to us and that in a way where both ISAR users and 
maintainers are happy. We do welcome any feedback
> ...
> And imagine yourself bumping a layer onto a new base-binary ... say
> after a buster-bullseye, NetworkManager->systemd-networkd,
> cron->systemd-timers jump ... good luck writing/reading that Changelog
I am sure there are corner cases that needs to be taken into 
consideration and that would definitely be part of the design exercise 
(remember this is just a "vision" document to seek alignment before 
anyone of us spends too much time). If the mechanism turns out to be 
more of a caching mechanism, you will then end-up with the same flow as 
today (i.e. all customized packages built from sources, upstream 
packages pull from their upstream repositories and the entire image 
built with the exact same mechanism as we have today)
>
> Henning
I hope this clarifies, the intent, the use-case and more importantly our 
desire to discuss before we go one direction or another or just halt.
>
> Am Tue, 1 Dec 2020 08:59:20 +0100
> schrieb Cedric Hombourger <Cedric_Hombourger@mentor.com>:
>
>> Image Templates for ISAR
>> ========================
>>
>> Introduction
>> ------------
>>
>> Embedded System projects may have multiple teams working at different
>> levels. It is not uncommon to have groups creating a "base platform"
>> which is handed over to product teams to customize further. Thanks to
>> ISAR leveraging bitbake, image recipes from a base layer may be
>> easily extended by product layers but also with its ability to
>> produce and use a `base-apt` instead of upstream package feeds. This
>> workflow functions well in many cases, it may not be optimal for
>> application teams. Such teams are typically receiving a base platform
>> release that is likely to be left untouched for weeks. Creating a
>> product image for those teams is sometimes as easy as adding a few
>> packages.
>>
>> In such cases, it may be desirable to have a mechanism for ISAR to
>> construct product images starting from the base platform root
>> file-system instead of ISAR's bootstrap root file-system. We will
>> refer to these base platform root file-systems as _Template Images_.
>>
>> This document will describe both the construction and the use of
>> _template images_.
>>
>> Construction
>> ------------
>>
>> Developers needing to produce image templates in addition to bootable
>> system images will have their image recipes `inherit` the
>> `produce-image-template` class either as part of their image
>> development or by extending an existing image with an appropriate
>> `.bbappend`.
>>
>> The class would invoke a `do_rootfs_template` task after
>> `do_rootfs_install` and before the post-processing tasks (which
>> should be applied on the final image). This new task would simply
>> create a template image in the deploy folder built from the common
>> root file-system artifacts. Some systems may have additional, more
>> extensive post-processing operations that must be performed on the
>> final image, this class will still allow for such operations to
>> function normally.
>>
>> It is anticipated that configuration files in
>> `/etc/apt/sources.list.d` and `/etc/resolv.conf` should be excluded
>> (they should be recreated while the template image is imported). A
>> subsequent update will define a criteria for determining which files
>> should be excluded from the tarball. As yet the specific mechanism
>> has not been determined, whether it would be a file list, pattern or
>> some other external attributes. We anticipate other objects that may
>> need to be excluded are `/etc/hosts` and proxy configuration, to name
>> two simple examples.
>>
>> The process of constructing the base image would then continue as
>> usual (with post-processing and imager tasks).
>>
>> It is suggested to expose the produced image template as an opaque
>> binary blob and use a `.tmpl.img` file extension although the initial
>> implementation is anticipated to be a compressed tarball. The
>> template filesystem may need embedded meta-data to be consumed by
>> import mechanism or image recipes that make use of templates but
>> omitted from the final rootfs. It would be useful at the outset to
>> either specify a minimal set of metadata (build date, image name and
>> feature list is the initial list to be considered) and the structure
>> of that metadata (currently considering either JSON-formatted
>> suitable for extraction with `jq` or `deb` packaging, which has
>> embedded metadata in the cpio archive).
>>
>> Usage
>> -----
>>
>> Application teams may create image recipes referencing image
>> templates in their SRC_URI list. As noted above, image templates
>> would use the `.tmpl.img` file extension. This would be used by
>> ISAR's `rootfs` class to locate image templates in the list and make
>> sure that only one template is specified (extracting a template over
>> another would have unpredictable results).
>>
>> If the bulk of the template intelligence is present in the
>> `do_rootfs` stages, the `do_rootfs_prepare` task from the `rootfs`
>> class would behave differently.  In this case, an image template may
>> be added to an image SRC_URI. Instead of copying ISAR's bootstrap
>> rootfs, the template feature would instead unpack the specified
>> template. It would also be necessary to copy
>> `/etc/apt/sources.list.d` from the bootstrap and possibly other files
>> (e.g. `/etc/resolv.conf`). As the environment may include different
>> apt sources, a complete `apt-get update` would be in order.
>>
>> ISAR would then "enter" the unpacked template carrying updated apt
>> sources to install packages listed in `IMAGE_INSTALL` and
>> `IMAGE_PREINSTALL` using `apt- get install` as it does today in
>> `do_rootfs_install`.
>>
>> This design should allow images constructed from an image template to
>> produce a new image template (e.g. a base platform could provide a
>> template having a BSP and a minimal Debian system, and middleware
>> teams could then produce their own template with middleware stacks
>> added and finally consumed by application teams to produce a final
>> system image).
>>
>> Future Work
>> -----------
>>
>> It may be desirable to offer a mechanism to remove packages coming
>> from an image template (application teams may need to optimize the
>> final system image to meet the footprint constraints of their design
>> and/or to reduce the attack surface by removing services they do not
>> need).
>>
>> Status
>> ------
>>
>> This high-level design was created to seek early feedback from the
>> ISAR developer community by exposing the problem we are trying to
>> solve and the mechanism we are proposing as a solution. The next step
>> would be the development of a Proof-of-Concept in the event where
>> positive comments are received. This document may also be updated as
>> feedback is received and may result in different approaches being
>> discussed.
>>
>> We will be happy to answer questions you may have or provide more
>> details in areas where this document would be too evasive.
>>

Re: [RFC] Image templates

[Claudius Heine]

Hi Cedric,

On 2020-12-01 08:59, Cedric Hombourger wrote:
> Image Templates for ISAR
> ========================
> 
> Introduction
> ------------
> 
> Embedded System projects may have multiple teams working at different 
> levels. It is not uncommon to have groups creating a "base platform" 
> which is handed over to product teams to customize further.
> Thanks to ISAR leveraging bitbake, image recipes from a base layer may 
> be easily extended by product layers but also with its ability to 
> produce and use a `base-apt` instead of upstream package feeds.
> This workflow functions well in many cases, it may not be optimal for 
> application teams. Such teams are typically receiving a base platform 
> release that is likely to be left untouched for weeks.
> Creating a product image for those teams is sometimes as easy as adding 
> a few packages.
> 
> In such cases, it may be desirable to have a mechanism for ISAR to 
> construct product images starting from the base platform root 
> file-system instead of ISAR's bootstrap root file-system. We will refer
> to these base platform root file-systems as _Template Images_.

In principle I understand your requirement, however I don't think Isar 
(and possible Debian) is the right base for this.

What you probably require is some sort of functional and non-destructive 
root-fs build approach to root file system, that lets you arbitrarily 
cache and distribute each stage of the root fs build. Generating and 
reusing the 'base-apt' repository is much easier than it would be for 
root file systems.

Also stuff like docker solves only a small part of that solution.

You will probably have to build a new system inspired by stuff like the 
bazel build system[1], and nixox[2] or guixos[3] and maybe even stuff 
like ostree [4].

The rest of what you describe sounds like they would try to shoehorn 
this into the existing isar build process and then trying to explicitly 
fix just some of the many future issues you will run into, instead of 
actually designing something that is actually able to deal with all 
those issues in a generic way.

Personally, I would be interested in such a new system, but that will 
take serious dedication to design and implement it properly.

Changing that now in Isar will probably break a lot of assumptions and 
therefore limit its usefulness.

This suggestions reminds me a bit as the eSDK of OE, where I still 
couldn't really find a use for and from my experience creates more 
issues than it actually solves.

regards,
Claudius

[1] https://www.bazel.build/
[2] https://nixos.org/
[3] https://guix.gnu.org/
[4] https://ostreedev.github.io/ostree/introduction/

Re: [RFC] Image templates

[Joe MacDonald]

[-- Attachment #1: Type: text/plain, Size: 10845 bytes --]

Hi Jan,

[Re: [RFC] Image templates] On 20.12.01 (Tue 09:40) Jan Kiszka wrote:

> On 01.12.20 08:59, Cedric Hombourger wrote:
> > Image Templates for ISAR
> > ========================
> > 
> > Introduction
> > ------------
> > 
> > Embedded System projects may have multiple teams working at different
> > levels. It is not uncommon to have groups creating a "base platform"
> > which is handed over to product teams to customize further.
> > Thanks to ISAR leveraging bitbake, image recipes from a base layer may
> > be easily extended by product layers but also with its ability to
> > produce and use a `base-apt` instead of upstream package feeds.
> > This workflow functions well in many cases, it may not be optimal for
> > application teams. Such teams are typically receiving a base platform
> > release that is likely to be left untouched for weeks.
> > Creating a product image for those teams is sometimes as easy as adding
> > a few packages.
> 
> There is another use case, even for base image developers: accelerating
> the turn-around time when working only on few, widely isolated packages,
> may it be the kernel or a bootloader artifact.
> 
> For that scenario, it should be possible to define an image template
> excluding those bits, only injecting them on top in a second step.

That's a very good point.  We will incorporate the additional use case
idea in a revised verison of the proposal.  Do you have thoughts on what
the implementation may look like for this?  An explicit list of packages
in the image that should be removed or otherwise excluded from the
generated template?

> > In such cases, it may be desirable to have a mechanism for ISAR to
> > construct product images starting from the base platform root
> > file-system instead of ISAR's bootstrap root file-system. We will refer
> > to these base platform root file-systems as _Template Images_.
> > 
> > This document will describe both the construction and the use of
> > _template images_.
> > 
> > Construction
> > ------------
> > 
> > Developers needing to produce image templates in addition to bootable
> > system images will have their image recipes `inherit` the
> > `produce-image-template` class either as part of their image development
> > or by extending an existing image with an appropriate `.bbappend`.
> > 
> > The class would invoke a `do_rootfs_template` task after
> > `do_rootfs_install` and before the post-processing tasks (which should
> > be applied on the final image). This new task would simply create a
> > template image in the deploy folder built from the common root
> > file-system artifacts. Some systems may have additional, more extensive
> > post-processing operations that must be performed on the final
> > image, this class will still allow for such operations to function
> > normally.
> > 
> > It is anticipated that configuration files in `/etc/apt/sources.list.d`
> > and `/etc/resolv.conf` should be excluded (they should be recreated
> > while the template image is imported). A subsequent update
> > will define a criteria for determining which files should be excluded
> > from the tarball. As yet the specific mechanism has not been determined,
> > whether it would be a file list, pattern or some other
> > external attributes. We anticipate other objects that may need to be
> > excluded are `/etc/hosts` and proxy configuration, to name two simple
> > examples.
> > 
> > The process of constructing the base image would then continue as usual
> > (with post-processing and imager tasks).
> > 
> > It is suggested to expose the produced image template as an opaque
> > binary blob and use a `.tmpl.img` file extension although the initial
> > implementation is anticipated to be a compressed tarball.
> > The template filesystem may need embedded meta-data to be consumed by
> > import mechanism or image recipes that make use of templates but omitted
> > from the final rootfs. It would be useful at the outset
> > to either specify a minimal set of metadata (build date, image name and
> > feature list is the initial list to be considered) and the structure of
> > that metadata (currently considering either JSON-formatted
> > suitable for extraction with `jq` or `deb` packaging, which has embedded
> > metadata in the cpio archive).
> > 
> > Usage
> > -----
> > 
> > Application teams may create image recipes referencing image templates
> > in their SRC_URI list. As noted above, image templates would use the
> > `.tmpl.img` file extension. This would be used by ISAR's `rootfs`
> > class to locate image templates in the list and make sure that only one
> > template is specified (extracting a template over another would have
> > unpredictable results).
> > 
> > If the bulk of the template intelligence is present in the `do_rootfs`
> > stages, the `do_rootfs_prepare` task from the `rootfs` class would
> > behave differently.  In this case, an image template may be added to
> > an image SRC_URI. Instead of copying ISAR's bootstrap rootfs, the
> > template feature would instead unpack the specified template. It would
> > also be necessary to copy `/etc/apt/sources.list.d` from the bootstrap
> > and possibly other files (e.g. `/etc/resolv.conf`). As the environment
> > may include different apt sources, a complete `apt-get update` would be
> > in order.
> > 
> > ISAR would then "enter" the unpacked template carrying updated apt
> > sources to install packages listed in `IMAGE_INSTALL` and
> > `IMAGE_PREINSTALL` using `apt- get install` as it does today in
> > `do_rootfs_install`.
> > 
> > This design should allow images constructed from an image template to
> > produce a new image template (e.g. a base platform could provide a
> > template having a BSP and a minimal Debian system, and middleware teams
> > could then produce their own template with middleware stacks added and
> > finally consumed by application teams to produce a final system image).
> > 
> 
> There will likely be a number of traps we can stumble over, but the
> general vision is shared.
> 
> One of the trap is that you may not only need a target image template
> but also the corresponding buildchroots, if not isar-bootstraps.

Also a good point.  I wonder if we could avoid this by having a second
manifest (maybe another SRC_URI-like variable) for the template which
would include development versions of all installed packages or something
and then during development the template will include them but the final
image that derives from the template would have those removed unless
specifically added by the final image SRC_URI.  It would be nice, if
possible, to avoid having to bundle the image and the chroot/bootstrap
directories as well.

> Would your model be that a developer works within the layer that creates
> the template? A key issue would be that changes to the layer that
> invalidate the template would be detected and make the template-based
> build fail or fall back to the full build.
> 
> Or would your model be that templating enforce a split build via
> separate repos? That might be problematic from the maintenance
> perspective as application injection will come before imaging, and the
> latter is not the task of an application developer.

Cedric and I may have different views on this.  Personally I had been
thinking it would be more along the lines of the latter, where the
templates are entirely an external entity that a developer brings in, so
there would be no 'risk' of the template being invalidated and recreating
a full build.  The primary motiviation for this, from my perspective, is
that it's better for such a change to cause a failure and be flagged
appropriately than for it to silently build and give the developer the
impression that they are building on top of the specified template but are
really using something different.

I'm not sure what you mean about this being a problem for application
developers.  Can you elaborate a bit?  It seems to me that this isn't
significantly different thatn any existing Isar workflow is today and it
is still possible for the application developer to simply generate a
binary deb and install that on a deployed system whether that system was
assembled from templates or in the more traditional manner.

> > Future Work
> > -----------
> > 
> > It may be desirable to offer a mechanism to remove packages coming from
> > an image template (application teams may need to optimize the final
> > system image to meet the footprint constraints of their design and/or
> > to reduce the attack surface by removing services they do not need).
> > 
> > Status
> > ------
> > 
> > This high-level design was created to seek early feedback from the ISAR
> > developer community by exposing the problem we are trying to solve and
> > the mechanism we are proposing as a solution. The next step would be
> > the development of a Proof-of-Concept in the event where positive
> > comments are received. This document may also be updated as feedback is
> > received and may result in different approaches being discussed.
> > 
> > We will be happy to answer questions you may have or provide more
> > details in areas where this document would be too evasive.
> > 
> 
> The whole thing also reminds me a lot of how docker images are created.
> I wonder to which degree we can learn from that, reuse concepts or avoid
> pitfalls. Adding Silvano with whom I was recently discussing base
> container image generation via Isar.

Oh, that's very interesting!  I'd been thinking the same thing, and
certainly it's no coincidence that the proposal does somewhat follow the
same model as Docker container assembly.  I think there are some very good
ideas in the way Docker containers are created and there are some things
that are very limiting and seem like unnecessary restrictions, so I would
certainly welcome the opportunity to discuss those further.

About a year ago I'd started putting infrastructure into our IndustrialOS
builds to automatically generate Docker images alongside our current IPC
wic images.  In the end, coming up with a good way to create usable
Docker-style workflows seemed like more effort than we could justify at
the time for something for which we had no customer-driven use-case, and I
just created a script to do much the same thing outside of the Isar build
process.  I still think it's an excellent idea to use Isar to generate
base images, though, I'd be very interested in the discussion when you're
ready to share details.

> 
> Jan
> 
> -- 
> Siemens AG, T RDA IOT
> Corporate Competence Center Embedded Linux

-- 
-Joe MacDonald.
Linux Architect | Mentor® A Siemens Business
:wq

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [RFC] Image templates

[Joe MacDonald]

[-- Attachment #1: Type: text/plain, Size: 12775 bytes --]

Hi Jan,

One quick follow-up question on your vision here.  As I am revising our
document it's occurred to me that my original thought about this is
probably flawed:

> > There is another use case, even for base image developers: accelerating
> > the turn-around time when working only on few, widely isolated packages,
> > may it be the kernel or a bootloader artifact.
> > 
> > For that scenario, it should be possible to define an image template
> > excluding those bits, only injecting them on top in a second step.
> 
> That's a very good point.  We will incorporate the additional use case
> idea in a revised verison of the proposal.  Do you have thoughts on what
> the implementation may look like for this?  An explicit list of packages
> in the image that should be removed or otherwise excluded from the
> generated template?

I had thought this would be something to add to the template generation
functionality, and maybe that's still the right way to do it, but this
somewhat breaks the clean separation of producers and consumers in this
template model.  I think what we want is for the 'exclude' part to be
something the application developer explicitly states when incorporating a
template, probably in thier layer.  Do you agree?  Have a different idea
how we would approach this?

Thanks,
-Joe.

[Re: [RFC] Image templates] On 20.12.02 (Wed 09:42) Joe MacDonald wrote:

> Hi Jan,
> 
> [Re: [RFC] Image templates] On 20.12.01 (Tue 09:40) Jan Kiszka wrote:
> 
> > On 01.12.20 08:59, Cedric Hombourger wrote:
> > > Image Templates for ISAR
> > > ========================
> > > 
> > > Introduction
> > > ------------
> > > 
> > > Embedded System projects may have multiple teams working at different
> > > levels. It is not uncommon to have groups creating a "base platform"
> > > which is handed over to product teams to customize further.
> > > Thanks to ISAR leveraging bitbake, image recipes from a base layer may
> > > be easily extended by product layers but also with its ability to
> > > produce and use a `base-apt` instead of upstream package feeds.
> > > This workflow functions well in many cases, it may not be optimal for
> > > application teams. Such teams are typically receiving a base platform
> > > release that is likely to be left untouched for weeks.
> > > Creating a product image for those teams is sometimes as easy as adding
> > > a few packages.
> > 
> > There is another use case, even for base image developers: accelerating
> > the turn-around time when working only on few, widely isolated packages,
> > may it be the kernel or a bootloader artifact.
> > 
> > For that scenario, it should be possible to define an image template
> > excluding those bits, only injecting them on top in a second step.
> 
> That's a very good point.  We will incorporate the additional use case
> idea in a revised verison of the proposal.  Do you have thoughts on what
> the implementation may look like for this?  An explicit list of packages
> in the image that should be removed or otherwise excluded from the
> generated template?
> 
> > > In such cases, it may be desirable to have a mechanism for ISAR to
> > > construct product images starting from the base platform root
> > > file-system instead of ISAR's bootstrap root file-system. We will refer
> > > to these base platform root file-systems as _Template Images_.
> > > 
> > > This document will describe both the construction and the use of
> > > _template images_.
> > > 
> > > Construction
> > > ------------
> > > 
> > > Developers needing to produce image templates in addition to bootable
> > > system images will have their image recipes `inherit` the
> > > `produce-image-template` class either as part of their image development
> > > or by extending an existing image with an appropriate `.bbappend`.
> > > 
> > > The class would invoke a `do_rootfs_template` task after
> > > `do_rootfs_install` and before the post-processing tasks (which should
> > > be applied on the final image). This new task would simply create a
> > > template image in the deploy folder built from the common root
> > > file-system artifacts. Some systems may have additional, more extensive
> > > post-processing operations that must be performed on the final
> > > image, this class will still allow for such operations to function
> > > normally.
> > > 
> > > It is anticipated that configuration files in `/etc/apt/sources.list.d`
> > > and `/etc/resolv.conf` should be excluded (they should be recreated
> > > while the template image is imported). A subsequent update
> > > will define a criteria for determining which files should be excluded
> > > from the tarball. As yet the specific mechanism has not been determined,
> > > whether it would be a file list, pattern or some other
> > > external attributes. We anticipate other objects that may need to be
> > > excluded are `/etc/hosts` and proxy configuration, to name two simple
> > > examples.
> > > 
> > > The process of constructing the base image would then continue as usual
> > > (with post-processing and imager tasks).
> > > 
> > > It is suggested to expose the produced image template as an opaque
> > > binary blob and use a `.tmpl.img` file extension although the initial
> > > implementation is anticipated to be a compressed tarball.
> > > The template filesystem may need embedded meta-data to be consumed by
> > > import mechanism or image recipes that make use of templates but omitted
> > > from the final rootfs. It would be useful at the outset
> > > to either specify a minimal set of metadata (build date, image name and
> > > feature list is the initial list to be considered) and the structure of
> > > that metadata (currently considering either JSON-formatted
> > > suitable for extraction with `jq` or `deb` packaging, which has embedded
> > > metadata in the cpio archive).
> > > 
> > > Usage
> > > -----
> > > 
> > > Application teams may create image recipes referencing image templates
> > > in their SRC_URI list. As noted above, image templates would use the
> > > `.tmpl.img` file extension. This would be used by ISAR's `rootfs`
> > > class to locate image templates in the list and make sure that only one
> > > template is specified (extracting a template over another would have
> > > unpredictable results).
> > > 
> > > If the bulk of the template intelligence is present in the `do_rootfs`
> > > stages, the `do_rootfs_prepare` task from the `rootfs` class would
> > > behave differently.  In this case, an image template may be added to
> > > an image SRC_URI. Instead of copying ISAR's bootstrap rootfs, the
> > > template feature would instead unpack the specified template. It would
> > > also be necessary to copy `/etc/apt/sources.list.d` from the bootstrap
> > > and possibly other files (e.g. `/etc/resolv.conf`). As the environment
> > > may include different apt sources, a complete `apt-get update` would be
> > > in order.
> > > 
> > > ISAR would then "enter" the unpacked template carrying updated apt
> > > sources to install packages listed in `IMAGE_INSTALL` and
> > > `IMAGE_PREINSTALL` using `apt- get install` as it does today in
> > > `do_rootfs_install`.
> > > 
> > > This design should allow images constructed from an image template to
> > > produce a new image template (e.g. a base platform could provide a
> > > template having a BSP and a minimal Debian system, and middleware teams
> > > could then produce their own template with middleware stacks added and
> > > finally consumed by application teams to produce a final system image).
> > > 
> > 
> > There will likely be a number of traps we can stumble over, but the
> > general vision is shared.
> > 
> > One of the trap is that you may not only need a target image template
> > but also the corresponding buildchroots, if not isar-bootstraps.
> 
> Also a good point.  I wonder if we could avoid this by having a second
> manifest (maybe another SRC_URI-like variable) for the template which
> would include development versions of all installed packages or something
> and then during development the template will include them but the final
> image that derives from the template would have those removed unless
> specifically added by the final image SRC_URI.  It would be nice, if
> possible, to avoid having to bundle the image and the chroot/bootstrap
> directories as well.
> 
> > Would your model be that a developer works within the layer that creates
> > the template? A key issue would be that changes to the layer that
> > invalidate the template would be detected and make the template-based
> > build fail or fall back to the full build.
> > 
> > Or would your model be that templating enforce a split build via
> > separate repos? That might be problematic from the maintenance
> > perspective as application injection will come before imaging, and the
> > latter is not the task of an application developer.
> 
> Cedric and I may have different views on this.  Personally I had been
> thinking it would be more along the lines of the latter, where the
> templates are entirely an external entity that a developer brings in, so
> there would be no 'risk' of the template being invalidated and recreating
> a full build.  The primary motiviation for this, from my perspective, is
> that it's better for such a change to cause a failure and be flagged
> appropriately than for it to silently build and give the developer the
> impression that they are building on top of the specified template but are
> really using something different.
> 
> I'm not sure what you mean about this being a problem for application
> developers.  Can you elaborate a bit?  It seems to me that this isn't
> significantly different thatn any existing Isar workflow is today and it
> is still possible for the application developer to simply generate a
> binary deb and install that on a deployed system whether that system was
> assembled from templates or in the more traditional manner.
> 
> > > Future Work
> > > -----------
> > > 
> > > It may be desirable to offer a mechanism to remove packages coming from
> > > an image template (application teams may need to optimize the final
> > > system image to meet the footprint constraints of their design and/or
> > > to reduce the attack surface by removing services they do not need).
> > > 
> > > Status
> > > ------
> > > 
> > > This high-level design was created to seek early feedback from the ISAR
> > > developer community by exposing the problem we are trying to solve and
> > > the mechanism we are proposing as a solution. The next step would be
> > > the development of a Proof-of-Concept in the event where positive
> > > comments are received. This document may also be updated as feedback is
> > > received and may result in different approaches being discussed.
> > > 
> > > We will be happy to answer questions you may have or provide more
> > > details in areas where this document would be too evasive.
> > > 
> > 
> > The whole thing also reminds me a lot of how docker images are created.
> > I wonder to which degree we can learn from that, reuse concepts or avoid
> > pitfalls. Adding Silvano with whom I was recently discussing base
> > container image generation via Isar.
> 
> Oh, that's very interesting!  I'd been thinking the same thing, and
> certainly it's no coincidence that the proposal does somewhat follow the
> same model as Docker container assembly.  I think there are some very good
> ideas in the way Docker containers are created and there are some things
> that are very limiting and seem like unnecessary restrictions, so I would
> certainly welcome the opportunity to discuss those further.
> 
> About a year ago I'd started putting infrastructure into our IndustrialOS
> builds to automatically generate Docker images alongside our current IPC
> wic images.  In the end, coming up with a good way to create usable
> Docker-style workflows seemed like more effort than we could justify at
> the time for something for which we had no customer-driven use-case, and I
> just created a script to do much the same thing outside of the Isar build
> process.  I still think it's an excellent idea to use Isar to generate
> base images, though, I'd be very interested in the discussion when you're
> ready to share details.
> 
> > 
> > Jan
> > 
> > -- 
> > Siemens AG, T RDA IOT
> > Corporate Competence Center Embedded Linux
> 
> -- 
> -Joe MacDonald.
> Linux Architect | Mentor® A Siemens Business
> :wq



-- 
-Joe MacDonald.
Linux Architect | Mentor® A Siemens Business
:wq

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [RFC] Image templates

[Joe MacDonald]

[-- Attachment #1: Type: text/plain, Size: 4913 bytes --]

Hi Claudius,

[Re: [RFC] Image templates] On 20.12.02 (Wed 14:00) Claudius Heine wrote:

> Hi Cedric,
> 
> On 2020-12-01 08:59, Cedric Hombourger wrote:
> > Image Templates for ISAR
> > ========================
> > 
> > Introduction
> > ------------
> > 
> > Embedded System projects may have multiple teams working at different
> > levels. It is not uncommon to have groups creating a "base platform"
> > which is handed over to product teams to customize further.
> > Thanks to ISAR leveraging bitbake, image recipes from a base layer may
> > be easily extended by product layers but also with its ability to
> > produce and use a `base-apt` instead of upstream package feeds.
> > This workflow functions well in many cases, it may not be optimal for
> > application teams. Such teams are typically receiving a base platform
> > release that is likely to be left untouched for weeks.
> > Creating a product image for those teams is sometimes as easy as adding
> > a few packages.
> > 
> > In such cases, it may be desirable to have a mechanism for ISAR to
> > construct product images starting from the base platform root
> > file-system instead of ISAR's bootstrap root file-system. We will refer
> > to these base platform root file-systems as _Template Images_.
> 
> In principle I understand your requirement, however I don't think Isar (and
> possible Debian) is the right base for this.
> 
> What you probably require is some sort of functional and non-destructive
> root-fs build approach to root file system, that lets you arbitrarily cache
> and distribute each stage of the root fs build. Generating and reusing the
> 'base-apt' repository is much easier than it would be for root file systems.
> 
> Also stuff like docker solves only a small part of that solution.
> 
> You will probably have to build a new system inspired by stuff like the
> bazel build system[1], and nixox[2] or guixos[3] and maybe even stuff like
> ostree [4].

Thank you for the suggestions.  I had looked at bazel a while back for
something unrelated but my impression of it is really that it's aimed
squarely at application development teams, so never really investigated
how they do their builds.  We had a proof-of-concept working about a year
ago with ostree on top of Isar but ultimately that didn't seem to be a
good fit either since it placed significant restrictions on the
filesystem.  In the intervening time the project has also moved in a more
container-focused direction, so it also doesn't seem to address this space
any longer.

NixOS and Guix both warrant further investigation, I think, though I've
done a bit of preliminary work with NixOS and I think the differences
between it and Debian in package format and maintenance tools are too
great to make it practical.

In short, I definitely agree this is potentially a significant adjustment
and a large undertaking, but Isar does also seem almost uniquely
positioned to make this work, and why whe're making this proposal and
looking for feedback.

> The rest of what you describe sounds like they would try to shoehorn this
> into the existing isar build process and then trying to explicitly fix just
> some of the many future issues you will run into, instead of actually
> designing something that is actually able to deal with all those issues in a
> generic way.

Are you thinking of something specific in the design that you see being
long-term problematic, or is this more of a general concern like "this is
plainly out of scope for Isar"?  Our objective is to produce something
that compliments and extends existing workflows, not to bolt on something
that doesn't make sense, so we're open to all feedback and views of what
could be danger areas or sources for future pain.

> Personally, I would be interested in such a new system, but that will take
> serious dedication to design and implement it properly.
> 
> Changing that now in Isar will probably break a lot of assumptions and
> therefore limit its usefulness.

That's definitley not our goal and the approach we are proposing, done as
a new class, intends to ensure it doesn't spider out and touch things that
will impact unrelated Isar workflows.  If you can point to areas where
we're potentially breaking assumptions in Isar, most likely around image
assembly, but maybe other oareas, we'd be very happy to refine our view
and goals.

Thanks for the feedback,
-Joe.

> This suggestions reminds me a bit as the eSDK of OE, where I still couldn't
> really find a use for and from my experience creates more issues than it
> actually solves.
> 
> regards,
> Claudius
> 
> [1] https://www.bazel.build/
> [2] https://nixos.org/
> [3] https://guix.gnu.org/
> [4] https://ostreedev.github.io/ostree/introduction/

-- 
-Joe MacDonald.
Linux Architect | Mentor® A Siemens Business
:wq

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [RFC] Image templates

[Claudius Heine]

Hi Joe,

On 2020-12-02 17:37, Joe MacDonald wrote:
> Hi Claudius,
> 
> [Re: [RFC] Image templates] On 20.12.02 (Wed 14:00) Claudius Heine wrote:
> 
>> Hi Cedric,
>>
>> On 2020-12-01 08:59, Cedric Hombourger wrote:
>>> Image Templates for ISAR
>>> ========================
>>>
>>> Introduction
>>> ------------
>>>
>>> Embedded System projects may have multiple teams working at different
>>> levels. It is not uncommon to have groups creating a "base platform"
>>> which is handed over to product teams to customize further.
>>> Thanks to ISAR leveraging bitbake, image recipes from a base layer may
>>> be easily extended by product layers but also with its ability to
>>> produce and use a `base-apt` instead of upstream package feeds.
>>> This workflow functions well in many cases, it may not be optimal for
>>> application teams. Such teams are typically receiving a base platform
>>> release that is likely to be left untouched for weeks.
>>> Creating a product image for those teams is sometimes as easy as adding
>>> a few packages.
>>>
>>> In such cases, it may be desirable to have a mechanism for ISAR to
>>> construct product images starting from the base platform root
>>> file-system instead of ISAR's bootstrap root file-system. We will refer
>>> to these base platform root file-systems as _Template Images_.
>>
>> In principle I understand your requirement, however I don't think Isar (and
>> possible Debian) is the right base for this.
>>
>> What you probably require is some sort of functional and non-destructive
>> root-fs build approach to root file system, that lets you arbitrarily cache
>> and distribute each stage of the root fs build. Generating and reusing the
>> 'base-apt' repository is much easier than it would be for root file systems.
>>
>> Also stuff like docker solves only a small part of that solution.
>>
>> You will probably have to build a new system inspired by stuff like the
>> bazel build system[1], and nixox[2] or guixos[3] and maybe even stuff like
>> ostree [4].
> 
> Thank you for the suggestions.  I had looked at bazel a while back for
> something unrelated but my impression of it is really that it's aimed
> squarely at application development teams, so never really investigated
> how they do their builds.

The list is more for inspiration sake, not to actually use any of those 
technologies.

You are right Bazel is not really useful for this kind of work. I also 
don't have much experience with this, but AFAIK they are more of a 
Copy-on-Write build system. Which might be a good idea, because that 
would allow you to share partial build products.

>  We had a proof-of-concept working about a year
> ago with ostree on top of Isar but ultimately that didn't seem to be a
> good fit either since it placed significant restrictions on the
> filesystem.  In the intervening time the project has also moved in a more
> container-focused direction, so it also doesn't seem to address this space
> any longer.

Again my point was not using ostree directly, but to let you be inspired 
by it. What they did was putting a version of root file systems, that 
lets you reuse parts and create small deltas.

> NixOS and Guix both warrant further investigation, I think, though I've
> done a bit of preliminary work with NixOS and I think the differences
> between it and Debian in package format and maintenance tools are too
> great to make it practical.

NixOS lets you download pre-build artifacts at the same time as allowing 
you to add custom patches to packages, causing it to just rebuild only 
what you changed, which would again be a pretty great attribut to have 
for such a system.

> In short, I definitely agree this is potentially a significant adjustment
> and a large undertaking, but Isar does also seem almost uniquely
> positioned to make this work, and why whe're making this proposal and
> looking for feedback.

Isar is based on Debian, and while Debian is a great general 
Distribution, I don't think that pick and choosing different rootfs 
templates with different versions together would be possible. Which 
would IMO be a central property for such a system.

I understand that this proposal only considers a multiple images based 
on one template (1:m), because doing it otherwise could not really be 
solved here. But the modularity of bitbake based projects with OE and 
Isar is about m:n, e.g. multiple layers can be used and reused for 
multiple images.

>> The rest of what you describe sounds like they would try to shoehorn this
>> into the existing isar build process and then trying to explicitly fix just
>> some of the many future issues you will run into, instead of actually
>> designing something that is actually able to deal with all those issues in a
>> generic way.
> 
> Are you thinking of something specific in the design that you see being
> long-term problematic, or is this more of a general concern like "this is
> plainly out of scope for Isar"?  Our objective is to produce something
> that compliments and extends existing workflows, not to bolt on something
> that doesn't make sense, so we're open to all feedback and views of what
> could be danger areas or sources for future pain.

You will have to deal also with removal of packages that where installed 
in a template, and possible allow to patch and rebuild packages that are 
part of the template. That would mean the packages written in other all 
other isar layers now could cause a whole new dimension of issues. Isar 
packages don't really have the same QA as packages that are part of 
Debian and might not work well with removal. (This is also pointed out 
by Henning)

No idea how you would deal with detecting if some packages needs to be 
rebuild, because a patch was added. You will probably have to deal with 
signatures and sstate..

> 
>> Personally, I would be interested in such a new system, but that will take
>> serious dedication to design and implement it properly.
>>
>> Changing that now in Isar will probably break a lot of assumptions and
>> therefore limit its usefulness.
> 
> That's definitley not our goal and the approach we are proposing, done as
> a new class, intends to ensure it doesn't spider out and touch things that
> will impact unrelated Isar workflows.  If you can point to areas where
> we're potentially breaking assumptions in Isar, most likely around image
> assembly, but maybe other oareas, we'd be very happy to refine our view
> and goals.

- Packages in Isar don't have the same QA as packages in Debian, and 
might not work with removal, because the assumption was that if they 
should not be part of an image, they would not have been installed.
- Packages that are part of the template might need to be recompiled, 
because the image itself installed a different version of a library. 
This is solved in isar, because it would just build everything and let 
apt figure out the correct dependencies and installs all packages, but 
with this, you don't really have a way to call back into isar to compile 
some more packages. (e.g. kernel modules that are part of the template 
image, and I make a small, stable kernel update in the depending image)

I would expect a lot more of those...

regards,
Claudius

Re: [RFC] Image templates

[Jan Kiszka]

On 03.12.20 13:23, Claudius Heine wrote:
> Hi Joe,
> 
> On 2020-12-02 17:37, Joe MacDonald wrote:
>> Hi Claudius,
>>
>> [Re: [RFC] Image templates] On 20.12.02 (Wed 14:00) Claudius Heine wrote:
>>
>>> Hi Cedric,
>>>
>>> On 2020-12-01 08:59, Cedric Hombourger wrote:
>>>> Image Templates for ISAR
>>>> ========================
>>>>
>>>> Introduction
>>>> ------------
>>>>
>>>> Embedded System projects may have multiple teams working at different
>>>> levels. It is not uncommon to have groups creating a "base platform"
>>>> which is handed over to product teams to customize further.
>>>> Thanks to ISAR leveraging bitbake, image recipes from a base layer may
>>>> be easily extended by product layers but also with its ability to
>>>> produce and use a `base-apt` instead of upstream package feeds.
>>>> This workflow functions well in many cases, it may not be optimal for
>>>> application teams. Such teams are typically receiving a base platform
>>>> release that is likely to be left untouched for weeks.
>>>> Creating a product image for those teams is sometimes as easy as adding
>>>> a few packages.
>>>>
>>>> In such cases, it may be desirable to have a mechanism for ISAR to
>>>> construct product images starting from the base platform root
>>>> file-system instead of ISAR's bootstrap root file-system. We will refer
>>>> to these base platform root file-systems as _Template Images_.
>>>
>>> In principle I understand your requirement, however I don't think
>>> Isar (and
>>> possible Debian) is the right base for this.
>>>
>>> What you probably require is some sort of functional and non-destructive
>>> root-fs build approach to root file system, that lets you arbitrarily
>>> cache
>>> and distribute each stage of the root fs build. Generating and
>>> reusing the
>>> 'base-apt' repository is much easier than it would be for root file
>>> systems.
>>>
>>> Also stuff like docker solves only a small part of that solution.
>>>
>>> You will probably have to build a new system inspired by stuff like the
>>> bazel build system[1], and nixox[2] or guixos[3] and maybe even stuff
>>> like
>>> ostree [4].
>>
>> Thank you for the suggestions.  I had looked at bazel a while back for
>> something unrelated but my impression of it is really that it's aimed
>> squarely at application development teams, so never really investigated
>> how they do their builds.
> 
> The list is more for inspiration sake, not to actually use any of those
> technologies.
> 
> You are right Bazel is not really useful for this kind of work. I also
> don't have much experience with this, but AFAIK they are more of a
> Copy-on-Write build system. Which might be a good idea, because that
> would allow you to share partial build products.
> 
>>  We had a proof-of-concept working about a year
>> ago with ostree on top of Isar but ultimately that didn't seem to be a
>> good fit either since it placed significant restrictions on the
>> filesystem.  In the intervening time the project has also moved in a more
>> container-focused direction, so it also doesn't seem to address this
>> space
>> any longer.
> 
> Again my point was not using ostree directly, but to let you be inspired
> by it. What they did was putting a version of root file systems, that
> lets you reuse parts and create small deltas.
> 
>> NixOS and Guix both warrant further investigation, I think, though I've
>> done a bit of preliminary work with NixOS and I think the differences
>> between it and Debian in package format and maintenance tools are too
>> great to make it practical.
> 
> NixOS lets you download pre-build artifacts at the same time as allowing
> you to add custom patches to packages, causing it to just rebuild only
> what you changed, which would again be a pretty great attribut to have
> for such a system.
> 
>> In short, I definitely agree this is potentially a significant adjustment
>> and a large undertaking, but Isar does also seem almost uniquely
>> positioned to make this work, and why whe're making this proposal and
>> looking for feedback.
> 
> Isar is based on Debian, and while Debian is a great general
> Distribution, I don't think that pick and choosing different rootfs
> templates with different versions together would be possible. Which
> would IMO be a central property for such a system.
> 
> I understand that this proposal only considers a multiple images based
> on one template (1:m), because doing it otherwise could not really be
> solved here. But the modularity of bitbake based projects with OE and
> Isar is about m:n, e.g. multiple layers can be used and reused for
> multiple images.
> 
>>> The rest of what you describe sounds like they would try to shoehorn
>>> this
>>> into the existing isar build process and then trying to explicitly
>>> fix just
>>> some of the many future issues you will run into, instead of actually
>>> designing something that is actually able to deal with all those
>>> issues in a
>>> generic way.
>>
>> Are you thinking of something specific in the design that you see being
>> long-term problematic, or is this more of a general concern like "this is
>> plainly out of scope for Isar"?  Our objective is to produce something
>> that compliments and extends existing workflows, not to bolt on something
>> that doesn't make sense, so we're open to all feedback and views of what
>> could be danger areas or sources for future pain.
> 
> You will have to deal also with removal of packages that where installed
> in a template, and possible allow to patch and rebuild packages that are
> part of the template. That would mean the packages written in other all
> other isar layers now could cause a whole new dimension of issues. Isar
> packages don't really have the same QA as packages that are part of
> Debian and might not work well with removal. (This is also pointed out
> by Henning)
> 
> No idea how you would deal with detecting if some packages needs to be
> rebuild, because a patch was added. You will probably have to deal with
> signatures and sstate..
> 
>>
>>> Personally, I would be interested in such a new system, but that will
>>> take
>>> serious dedication to design and implement it properly.
>>>
>>> Changing that now in Isar will probably break a lot of assumptions and
>>> therefore limit its usefulness.
>>
>> That's definitley not our goal and the approach we are proposing, done as
>> a new class, intends to ensure it doesn't spider out and touch things
>> that
>> will impact unrelated Isar workflows.  If you can point to areas where
>> we're potentially breaking assumptions in Isar, most likely around image
>> assembly, but maybe other oareas, we'd be very happy to refine our view
>> and goals.
> 
> - Packages in Isar don't have the same QA as packages in Debian, and
> might not work with removal, because the assumption was that if they
> should not be part of an image, they would not have been installed.
> - Packages that are part of the template might need to be recompiled,
> because the image itself installed a different version of a library.
> This is solved in isar, because it would just build everything and let
> apt figure out the correct dependencies and installs all packages, but
> with this, you don't really have a way to call back into isar to compile
> some more packages. (e.g. kernel modules that are part of the template
> image, and I make a small, stable kernel update in the depending image)
> 
> I would expect a lot more of those...

I think a key element of any templating concept is defining its scope -
and limits. This cannot become a generic solution for all the scenarios
we currently model with a single image build. I also do not think it
will work for rebuilding random packages that have complex dependencies
- that's already hairy with plain Isar (think of rebuilding an essential
package...).

Probably the best and the cleanest we can do is to define that it is a
post-imaging package addition or removal step, acting fully within the
Debian packaging world. That will require additional QA on self-built
packages to make them cleanly deinstall as Claudius pointed out, but
that has a value of its own.

Template-based image post-processing may also be seen like
cross-compiling: known to work is native package compilation, so this
should be the production mode (though I know that many images use it for
production now). Analogously, the image template post-processing
approach should not be used to generate production images (famous last
words).

And that brings us to the technical requirements on generating a
template and building on top of it:

 - bundle all needed artifacts (template image AND buildchroots)
 - write a recipe that takes those artifacts and act as an alternative
   dependency to dpkg*-based recipes
 - write a recipe that takes packages, self-built or repo-provided, and
   installs them into a template, just like you would on the running
   device (alternative imaging target)
 - introduce some control mode for Isar to activate all this, switching
   dependencies accordingly

/Maybe/ we can also hook after rootfs generation but before imaging,
rerunning wic or other image classes also on the template. But that
would mandate even more that the post-processing will run over the very
same layer (sha or whatever signature-anchored) as the template was
generated by. But I would start as simple as possible.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [RFC] Image templates

[Baurzhan Ismagulov]

Hello Cedric,

thanks for the early RFC and the constructive discussion.

On Tue, Dec 01, 2020 at 08:59:20AM +0100, Cedric Hombourger wrote:
> In such cases, it may be desirable to have a mechanism for ISAR to construct product images starting from the base platform root file-system instead of ISAR's bootstrap root file-system. We will refer
> to these base platform root file-systems as _Template Images_.

Do I understand correctly, one way might look like this:

* Use case: Speed up component developer workflow.

* Approach:

  * Reuse build artifacts (rootfs, buildchroot, etc.).

  * Split builds (artifact preparation and consumption).

Artifact reuse and developer efficiency are among Isar's goals. We've been
(lazily) looking at this from the developer and CI PoVs. One step was isar-apt,
although it has to be finished in this regard (with pristine sources, the
packages are built even if present in isar-apt). I think especially for
application developers, template images (for some reason, I always write "image
templates" first :) ) would be a major time saver. We wanted to have some
benchmarking first (e.g., download, bootstrap, package building, imaging) -- if
you have any (even rough) figures, please share.

>From the practical perspective, I'm for any small, quick, least-invasive and
conceptually complete approach that we could gather experience with and discuss
the next steps. One way could be e.g.:

* Deploy a template image and implement consuming it.

* Package recipes depend on the template image. The image can be provided by
  building, or as a binary. The former is the default. One can switch to the
  latter in the configuration.

* There is a way to check whether the image is up-to-date regardless of the
  current mode (e.g., a task).

* Document the whole cycle -- where the image is stored, how it is copied to
  the working copy, etc.

With this experience, we could add the template e.g. to isar-bootstrap, as
originally implemented by Claudius.


Regarding source package quality and license compliance, I think it's in the
best interest of the product owners to provide that and isn't related to the
proposal.


Regarding alternative tooling, I see it rather in somewhat other areas which
are also not related to the topic:

* Standalone base-apt tooling would be very handy for certain use cases. As the
  current base-apt works, there is currently not much pressure behind this.

* Returning to multistrapping would allow efficient essential package
  replacement. We had a discussion with debian-embedded people regarding a
  Python implementation.

* apt introspection -- IIRC, ELBE colleagues use python-apt.

* Moving to pseudo instead of sudo -- Yocto is an example that this works. In
  general, we could clean up e.g. lazy umounts, replace bind mounts with cp
  -al, etc.

* From the very beginning, we were aware that bitbake provides only 80% of the
  functionality necessary for Debian (e.g., Build-Depends, building the
  pipeline after fetching). Meanwhile, we see ways how it could be improved, so
  we still haven't reached the bitbake vs. new tool discussion.


With kind regards,
Baurzhan.