From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6901194598360547328 X-Received: by 2002:a62:685:0:b029:198:d4a:5015 with SMTP id 127-20020a6206850000b02901980d4a5015mr1428071pfg.38.1606809580542; Mon, 30 Nov 2020 23:59:40 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a62:8f0c:: with SMTP id n12ls583861pfd.8.gmail; Mon, 30 Nov 2020 23:59:39 -0800 (PST) X-Google-Smtp-Source: ABdhPJzZFmmeCcEWKWsBzezdLwF28gMawTQc1JcyYwUkWCwNADaiJhWlrF87S/hxrcSC+BFfXpls X-Received: by 2002:a62:9202:0:b029:19b:40f0:5d40 with SMTP id o2-20020a6292020000b029019b40f05d40mr988930pfd.77.1606809579802; Mon, 30 Nov 2020 23:59:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1606809579; cv=none; d=google.com; s=arc-20160816; b=Xa18r+Ek3PureCnmOwL8jC2LECO30hojQsdI2MEtUVkkKjaB7Rav9Jtn1YxUE9Ek3i tcb8VCY3OlBSHXUgI0fY8TaESGv1hLcLuQgtZaGoLh68v9Ee6A20560zNLcGdeEO6dl6 2z6gCSkBjmTKCXnr3yscXleSxSA74ESpcFPVpF4W0KKiruC2DLbbK9Gfxrvv/8Jh8yVs f0iIYE+jyjtPL912LFxNyBYtd1qZHYzRFmd7kr1gzNlQHtIc36YU7I5TXvW9dkWeiOA8 ilVE51uwiG4X7rq1h8n9AdbVvABaB3OSsN6ohn1X3kH0B96L+WNhdGz+WG2vbwEUOpJO eFcg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-language:content-transfer-encoding:mime-version:user-agent :date:message-id:subject:from:cc:to:ironport-sdr:ironport-sdr; bh=/8l9HMrcbo8nfvQ49hrAtL0KwK8q7ciFAg4eszPuJX8=; b=A1Q68/4uGHnA4cgGTBQqEdb7KIhhxTwBviNFnQbuZ/fRaCR0YTaiy4LH0AASZ0NvTr TQ2n1Cft6vK0FaSGgcoQkyiv71xtFgrrlxgrR8I2C0TbKIac1rJPHVfIRt3Es+XoZWeJ GtkPiCz0s2hI8wLZRqhbrWn8JVxEaQ+pO7IdIHtMfBaiWQsiufvBy/NhMj9taE3E98Fs a7kRmm/N1XLDKl1+/Y2eS7sYTozWAgdoAHKu3Oz+emGS35hJLFXdA8nahHsW3Reaii9/ gWicKoiY87mzVOvng8msYzSN7EYICaI0Eh0ZcPrtunWvh17BkWbzfwXg5gG5bCPpWa8n 7uDQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of cedric_hombourger@mentor.com designates 68.232.129.153 as permitted sender) smtp.mailfrom=Cedric_Hombourger@mentor.com Return-Path: Received: from esa1.mentor.iphmx.com (esa1.mentor.iphmx.com. [68.232.129.153]) by gmr-mx.google.com with ESMTPS id e19si67840pgv.4.2020.11.30.23.59.39 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 30 Nov 2020 23:59:39 -0800 (PST) Received-SPF: pass (google.com: domain of cedric_hombourger@mentor.com designates 68.232.129.153 as permitted sender) client-ip=68.232.129.153; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of cedric_hombourger@mentor.com designates 68.232.129.153 as permitted sender) smtp.mailfrom=Cedric_Hombourger@mentor.com IronPort-SDR: ZFkMrrz/S/EpFYvwuFHXBrlJjggn/XN3ET6xrB8AwXj88gtNSeJlDjgo+rSaV97EyBqwzY1ywG wy37b9FOKkfpDvhN97PSmKwhGBxHG3mRDtzk1my1WuNOxCnT5P6vVY2TAGtbPXwcv7D9cx85vP zXLvPWEhgL7klodnt7npiz4LhPTVI8wcZsns5a1hzFxrKmyir7gezSzovdyt/VFnF7KzNjAQS+ iG+jYhyzctGvtNpzQk2D9wt91qk+lNkk+vqdZvNq62YH16H6E78iusu4VQQyE1oNyJzXFZeSwo Kro= X-IronPort-AV: E=Sophos;i="5.78,383,1599552000"; d="scan'208";a="57937666" Received: from orw-gwy-01-in.mentorg.com ([192.94.38.165]) by esa1.mentor.iphmx.com with ESMTP; 30 Nov 2020 23:59:39 -0800 IronPort-SDR: tUETxtQNhBLWXmlyCSNU+Wp4B5HuTN+zMjj511R1ErLwbbaMX4oVLbRbTwfPl6aGQVBMLcZSbT AiLOpD8UXpH+cBmLTcPsMC94Pxa/e2/5I94MoYBddwYca4hQ645mCvYwknualeqxapY1xrOiTr Hou3MMG9DU2JaIxGU4atocVZUEG9JZBosjnqSxoTBZXMwsWMBtWKZ/yaTBS1zRwWV5MPFKbe6q NJuue93be6yf/7ocvHNRdyraDKnaKn7++xkFD3Fx5ra3Nahc5HkTQGXbVRiJUL2AwVGdiiLn2m Ckk= To: CC: From: Cedric Hombourger Subject: [RFC] Image templates Message-ID: <6bf9a786-51b8-2430-90f5-2857746d003f@mentor.com> Date: Tue, 1 Dec 2020 08:59:20 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.5.0 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US Return-Path: Cedric_Hombourger@mentor.com X-Originating-IP: [137.202.0.90] X-ClientProxiedBy: svr-ies-mbx-09.mgc.mentorg.com (139.181.222.9) To svr-ies-mbx-02.mgc.mentorg.com (139.181.222.2) X-TUID: sb0KRASIuIIL Image Templates for ISAR ======================== Introduction ------------ Embedded System projects may have multiple teams working at different levels. It is not uncommon to have groups creating a "base platform" which is handed over to product teams to customize further. Thanks to ISAR leveraging bitbake, image recipes from a base layer may be easily extended by product layers but also with its ability to produce and use a `base-apt` instead of upstream package feeds. This workflow functions well in many cases, it may not be optimal for application teams. Such teams are typically receiving a base platform release that is likely to be left untouched for weeks. Creating a product image for those teams is sometimes as easy as adding a few packages. In such cases, it may be desirable to have a mechanism for ISAR to construct product images starting from the base platform root file-system instead of ISAR's bootstrap root file-system. We will refer to these base platform root file-systems as _Template Images_. This document will describe both the construction and the use of _template images_. Construction ------------ Developers needing to produce image templates in addition to bootable system images will have their image recipes `inherit` the `produce-image-template` class either as part of their image development or by extending an existing image with an appropriate `.bbappend`. The class would invoke a `do_rootfs_template` task after `do_rootfs_install` and before the post-processing tasks (which should be applied on the final image). This new task would simply create a template image in the deploy folder built from the common root file-system artifacts. Some systems may have additional, more extensive post-processing operations that must be performed on the final image, this class will still allow for such operations to function normally. It is anticipated that configuration files in `/etc/apt/sources.list.d` and `/etc/resolv.conf` should be excluded (they should be recreated while the template image is imported). A subsequent update will define a criteria for determining which files should be excluded from the tarball. As yet the specific mechanism has not been determined, whether it would be a file list, pattern or some other external attributes. We anticipate other objects that may need to be excluded are `/etc/hosts` and proxy configuration, to name two simple examples. The process of constructing the base image would then continue as usual (with post-processing and imager tasks). It is suggested to expose the produced image template as an opaque binary blob and use a `.tmpl.img` file extension although the initial implementation is anticipated to be a compressed tarball. The template filesystem may need embedded meta-data to be consumed by import mechanism or image recipes that make use of templates but omitted from the final rootfs. It would be useful at the outset to either specify a minimal set of metadata (build date, image name and feature list is the initial list to be considered) and the structure of that metadata (currently considering either JSON-formatted suitable for extraction with `jq` or `deb` packaging, which has embedded metadata in the cpio archive). Usage ----- Application teams may create image recipes referencing image templates in their SRC_URI list. As noted above, image templates would use the `.tmpl.img` file extension. This would be used by ISAR's `rootfs` class to locate image templates in the list and make sure that only one template is specified (extracting a template over another would have unpredictable results). If the bulk of the template intelligence is present in the `do_rootfs` stages, the `do_rootfs_prepare` task from the `rootfs` class would behave differently. In this case, an image template may be added to an image SRC_URI. Instead of copying ISAR's bootstrap rootfs, the template feature would instead unpack the specified template. It would also be necessary to copy `/etc/apt/sources.list.d` from the bootstrap and possibly other files (e.g. `/etc/resolv.conf`). As the environment may include different apt sources, a complete `apt-get update` would be in order. ISAR would then "enter" the unpacked template carrying updated apt sources to install packages listed in `IMAGE_INSTALL` and `IMAGE_PREINSTALL` using `apt- get install` as it does today in `do_rootfs_install`. This design should allow images constructed from an image template to produce a new image template (e.g. a base platform could provide a template having a BSP and a minimal Debian system, and middleware teams could then produce their own template with middleware stacks added and finally consumed by application teams to produce a final system image). Future Work ----------- It may be desirable to offer a mechanism to remove packages coming from an image template (application teams may need to optimize the final system image to meet the footprint constraints of their design and/or to reduce the attack surface by removing services they do not need). Status ------ This high-level design was created to seek early feedback from the ISAR developer community by exposing the problem we are trying to solve and the mechanism we are proposing as a solution. The next step would be the development of a Proof-of-Concept in the event where positive comments are received. This document may also be updated as feedback is received and may result in different approaches being discussed. We will be happy to answer questions you may have or provide more details in areas where this document would be too evasive.