[meta-isar] Proposal to improve initial device bootstrapping

["'Heinisch, Alexander' via isar-users" <isar-users@googlegroups.com>]

# Device Bootstrapping

This is a proposal to improve initial device bootstrapping with meta-isar by making the `isar-image-installer` a more versatile and general tool.

## Background

Currently, the `isar-image-installer` contains the target image to be deployed (copied) to the target device in it's root filesystem. The installer image has to be copied to a usb stick and executed on the target device.

In our current manufacturing setup we are targeting prebuilt devices without any OS precommissioned. Flashing images directly to disk is not possible at that stage easily.
That's why we are using the `isar-image-installer` to deploy the target images to the device via usb.

## Motivation

This approach works fine when working with a single device on desk, having keyboard and screen attached, but does not scale for large rollouts for multiple devices during manufacturing.

To scale that process I suggest not only supporting a usb stick scenario, but also a variant to boot via pxe boot (or ipxe-boot) into an live os (which could (and probably will) be `isar-image-installer`).

> Note: Currently, we are targeting x86 based architectures providing UEFI.

## Identified Problems

1.  **Problem**: The installer script has to provide an unattended mode.

    **Possible Solution**: Add setting for unattended mode either via well known config file or via kernel cmdline.

2.  **Problem**: When embedding the target image into the installer rootfs a rebuild of the installer image is required everytime we change the target image.

    **Possible Solution**: Installer image could download target image from http/ftp/s3 server at runtime and install it from memory. (Therefore, we have to ensure enough memory is provided, or probably support some kind of streaming functionality)

3.  **Problem**: Since pxe transferrs only the kernel and the initramfs via TFTP (rather slow) When using pxe we have to provide the rootfs of the installer via nfs.

    **Possible Solution**: Having an online installer downloading the target images from some external source, enables us to put all installer logic in the installers initramfs. Thus, no need for an installer-rootfs.

    > Note: This not always works. Since we also want to support the usb use case, loading the target image from rootfs is still a desireable option we have to maintain!

4.  **Problem**: Enrolling secure boot keys has to be done manually now. Currently we are using scripts to do so which get executed after the installer ran.
This is needed, since the installer is not signed.

    **Possible Solution**: Sign installer.

5.  **Problem**: Still, enrolling the keys manually upfront is cumbersome and error prone, and buying devices with preenrolled keys, oftentimes is not wanted due to additional cost and additional trust. Enrolling the keys after installation can be done, but again, is a manual task which should be automated.

    **Possible Solution**: Enroll secureboot keys as an additional step during installation.

    > Note: Since `installation` is not an appropriate term anymore, when not only the image get's installed but additional steps like key-enrollment takes place, I will call that workflow `target-bootstrapping` in the remainder of this text.

6.  **Problem**: Disc encryption is currently done on first boot of the device (detects if disk is already encrypted, and if not, encrypts it.) We saw that this process sometimes takes several minutes and is one of the crucial parts when initially starting up. In our scenario after a device got precommissioned it is put aside and stored (without initial boot of the target os). Once manufacturing needs to pick up a new device it is taken from there and assembled to the main asset shipped to the customer during asset production. Since that step has to be as easy and as fast as possible, waiting several minutes (due to initial encryption) to check basic device information or worse, failing at that stage is inacceptible.

    **Possible Solution**: Encrypt target device disks as an additional step during `target-bootstrapping`.

7.  **Problem**: After the initial procomissioning of the device status information of the device (e.g. serial number, hardware info) has to be transferred to our central mgmt. system.

    **Possible Solution**: Run custom scripts as part of the `target-bootstrapping`

8.  **Problem**: During `target-bootstrapping` the progress of the bootstrapping has to be visualized. When talking about bootstrapping multiple devices attaching a screen is not desired. Thus we plan to give some status indication via LED drivers as well, and also report status to our central mgmt. system.

    **Possible Solution**: Run custom scripts for status reporting. This means, that customizeable scripts shall be invoked before and after every single bootstrapping phase, and ideally also reporting an overall progress.

## Draft

Instead of excecuting the deploy image script as a systemd service we propose to implement a configurable target-bootstrapper, which takes prepackaged scripts as an input and invokes them in a generic way.

```
TARGET_BOOTSTRAPPER_ADDITIONAL_PACKAGES += " deploy-image"
TARGET_BOOTSTRAPPER_TASK_deploy-image[script] = "deploy-image-wic.sh"
TARGET_BOOTSTRAPPER_TASK_deploy-image[workdir] = "/usr/bin"
TARGET_BOOTSTRAPPER_TASK_deploy-image[effort] = "2"
```

This configuration enables us to reuse existing upstream (e.g. deploy-image [1]) as well as downstream scripts (e.g. encrypt partition [2] from cip-core or enroll secure boot keys from other downstream repo) without code-duplication.

To allow such bootstrapper to report progress between execution of each of the prepackaged scripts, customized status reporting utilities can be configured and will be invoked. Such utilities include e.g. led drivers, status reporting via an REST service, aso. 

Each script-configuration can not only specify a dedicated workdir and entrypoint, but also an effort-estimate to weight the work performed within a single script more accurately.

Besides coming up with an initial draft of such target-bootstrapping (will send a patchseries in the upcoming days) one of the first steps will be to refactor the existing deploy-image-wic.sh to allow for `unattended-mode` (based on this patche-series [3] from Jan Kiszka) and extend the script to support downloading the target images from an http server.


[1] https://github.com/ilbers/isar/blob/master/meta-isar/recipes-installer/deploy-image/files/deploy-image-wic.sh
[2] https://gitlab.com/cip-project/cip-core/isar-cip-core/-/blob/master/recipes-initramfs/initramfs-crypt-hook/files/encrypt_partition.script?ref_type=heads
[3] https://patchwork.isar-build.org/project/isar/patch/6279c4d497ade9a55cad9c0f2f21834ae97f964c.1719927511.git.jan.kiszka@siemens.com/

Looking forward for your inputs,
Thank you!
Alexander

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/AM7PR10MB33207E2BF4AD0A22E2A206B886A22%40AM7PR10MB3320.EURPRD10.PROD.OUTLOOK.COM.