* [PATCH v6 0/5] Add support for containerized root filesystems
@ 2021-03-09 20:52 Silvano Cirujano Cuesta
2021-03-09 20:52 ` [PATCH v6 1/5] classes: add root filesystem containerizing class Silvano Cirujano Cuesta
` (6 more replies)
0 siblings, 7 replies; 11+ messages in thread
From: Silvano Cirujano Cuesta @ 2021-03-09 20:52 UTC (permalink / raw)
To: isar-users
This patch series provides support for containerized root filesystems,
for both target images and SDKs.
For containerized target images the new image type `container-img` has
been added.
For containerized SDKs the task `populate_sdk` has been extended.
Containerized root filesystems are easy to distribute and run, enabling
this way following scenarios:
- Use ISAR to build container images meant to be run only in containers.
- Use the same ISAR configuration to build images for containers, VMs
and bare-metal.
- Easy SDK distribution and "installation".
- Quickly testing certain applications in the workstation using the
target root filesystem.
In order to build containerized target root filesystems `IMAGE_TYPE` has
to be `container-img`, additionally the container image format can be
selected with the variable `CONTAINER_FORMATS`. The default format is
`docker-archive`.
In order to build containerized SDKs the variable `SDK_FORMAT` has to
provide any of the supported container formats (e.g. `docker-archive`).
The default format is the legacy non-containerized: `tar_xz`.
It also provides a sample machine, multiconfigs and ci-testing.
More information about its usage is documented in the file
docs/user_manual.md.
A PoC/demo of this functionality (only the SDK part) has been created
based on the project https://github.com/siemens/meta-iot2050.
Jan Kiszka already tested and liked it! =>
https://github.com/siemens/meta-iot2050/issues/86#issuecomment-768907845
Successful builds of both containerized target and SDK are available on
the same PoC project:
- https://github.com/Silvanoc/meta-iot2050/actions/runs/558311580
- https://github.com/Silvanoc/meta-iot2050/actions/runs/558311581
and also the resulting images:
- https://github.com/users/Silvanoc/packages/container/package/meta-iot2050%2Fiot2050-debian-arm64
- https://github.com/users/Silvanoc/packages/container/package/meta-iot2050%2Fiot2050-debian-sdk-arm64
In order to get a feeling about its usage (you need Docker or Podman),
follow these simple copy&paste instructions:
https://github.com/Silvanoc/meta-iot2050/blob/master/kas/BUILDING-SDK-CONTAINER.md#running-the-sdk
Build instructions are available in the upper part of that document.
Two new dependencies (umoci and skopeo -backporting from bullseye to
buster works easily) are required to create containerized root
filesystems (as specified in the documentation).
Typical container image management actions (e.g. push an image to a
container image regitry) are out of scope. Available tools (Docker,
Skopeo, Buildah, Podman,...) should be used for these actions.
A patch will follow this one to get the dependencies into the container
images being provided by the project
https://github.com/siemens/kas (for `kas-container`, for example).
Silvano Cirujano Cuesta (5):
classes: add root filesystem containerizing class
classes: add new image type 'container-img'
sdk: add support for containerized sdk
docs: document creation of container images
ci: add container image testing configurations
doc/user_manual.md | 127 ++++++++++++++++++
meta-isar/conf/machine/container.conf | 5 +
.../conf/multiconfig/container-bullseye.conf | 4 +
.../conf/multiconfig/container-buster.conf | 4 +
.../conf/multiconfig/container-focal.conf | 4 +
.../conf/multiconfig/container-stretch.conf | 4 +
meta/classes/container-img.bbclass | 18 +++
.../classes/image-container-extension.bbclass | 81 +++++++++++
meta/classes/image-sdk-extension.bbclass | 42 +++++-
meta/classes/image.bbclass | 1 +
scripts/ci_build.sh | 11 +-
11 files changed, 293 insertions(+), 8 deletions(-)
create mode 100644 meta-isar/conf/machine/container.conf
create mode 100644 meta-isar/conf/multiconfig/container-bullseye.conf
create mode 100644 meta-isar/conf/multiconfig/container-buster.conf
create mode 100644 meta-isar/conf/multiconfig/container-focal.conf
create mode 100644 meta-isar/conf/multiconfig/container-stretch.conf
create mode 100644 meta/classes/container-img.bbclass
create mode 100644 meta/classes/image-container-extension.bbclass
--
2.30.1
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v6 1/5] classes: add root filesystem containerizing class
2021-03-09 20:52 [PATCH v6 0/5] Add support for containerized root filesystems Silvano Cirujano Cuesta
@ 2021-03-09 20:52 ` Silvano Cirujano Cuesta
2021-03-09 20:52 ` [PATCH v6 2/5] classes: add new image type 'container-img' Silvano Cirujano Cuesta
` (5 subsequent siblings)
6 siblings, 0 replies; 11+ messages in thread
From: Silvano Cirujano Cuesta @ 2021-03-09 20:52 UTC (permalink / raw)
To: isar-users
This class can be used to create container images which root filesystem
is that generated by the do_rootfs task.
Containerized root filesystems have following possible use-cases:
- Using ISAR as a container image builder.
- Simplify distribution of runtime rootfs (binaries, libraries,
configurations, ...) for application development or testing.
- Distributing SDKs.
Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>
---
.../classes/image-container-extension.bbclass | 81 +++++++++++++++++++
meta/classes/image.bbclass | 1 +
2 files changed, 82 insertions(+)
create mode 100644 meta/classes/image-container-extension.bbclass
diff --git a/meta/classes/image-container-extension.bbclass b/meta/classes/image-container-extension.bbclass
new file mode 100644
index 0000000..e26604a
--- /dev/null
+++ b/meta/classes/image-container-extension.bbclass
@@ -0,0 +1,81 @@
+# This software is a part of ISAR.
+# Copyright (C) Siemens AG, 2021
+#
+# SPDX-License-Identifier: MIT
+#
+# This class extends the image.bbclass for containerizing the root filesystem.
+
+CONTAINER_FORMATS ?= "docker-archive"
+
+containerize_rootfs() {
+ local cmd="/bin/dash"
+ local empty_tag="empty"
+ local full_tag="latest"
+ local oci_img_dir="${WORKDIR}/oci-image"
+ local rootfs="$1"
+ local rootfs_id="$2"
+ local container_formats="$3"
+
+ # prepare OCI container image skeleton
+ bbdebug 1 "prepare OCI container image skeleton"
+ rm -rf "${oci_img_dir}"
+ sudo umoci init --layout "${oci_img_dir}"
+ sudo umoci new --image "${oci_img_dir}:${empty_tag}"
+ sudo umoci config --image "${oci_img_dir}:${empty_tag}" \
+ --config.cmd="${cmd}"
+ sudo umoci unpack --image "${oci_img_dir}:${empty_tag}" \
+ "${oci_img_dir}_unpacked"
+
+ # add root filesystem as the flesh of the skeleton
+ sudo cp -a "${rootfs}"/* "${oci_img_dir}_unpacked/rootfs/"
+ # clean-up temporary files
+ sudo find "${oci_img_dir}_unpacked/rootfs/tmp" -mindepth 1 -delete
+
+ # pack container image
+ bbdebug 1 "pack container image"
+ sudo umoci repack --image "${oci_img_dir}:${full_tag}" \
+ "${oci_img_dir}_unpacked"
+ sudo umoci remove --image "${oci_img_dir}:${empty_tag}"
+ sudo rm -rf "${oci_img_dir}_unpacked"
+
+ # no root needed anymore
+ sudo chown --recursive $(id -u):$(id -g) "${oci_img_dir}"
+
+ # convert the OCI container image to the desired format
+ image_name="isar-${rootfs_id}"
+ for image_type in ${CONTAINER_FORMATS} ; do
+ image_archive="${DEPLOY_DIR_IMAGE}/${rootfs_id}-${image_type}.tar"
+ bbdebug 1 "Creating container image type: ${image_type}"
+ case "${image_type}" in
+ "docker-archive" | "oci-archive")
+ if [ "${image_type}" = "oci-archive" ] ; then
+ target="${image_type}:${image_archive}:latest"
+ else
+ target="${image_type}:${image_archive}:${image_name}:latest"
+ fi
+ rm -f "${image_archive}" "${image_archive}.xz"
+ bbdebug 2 "Converting OCI image to ${image_type}"
+ skopeo --insecure-policy copy \
+ "oci:${oci_img_dir}:${full_tag}" "${target}"
+ bbdebug 2 "Compressing image"
+ xz -T0 "${image_archive}"
+ ;;
+ "oci")
+ tar --create --xz --directory "${oci_img_dir}" \
+ --file "${image_archive}.xz" .
+ ;;
+ "docker-daemon" | "containers-storage")
+ if [ -f /.dockerenv ] || [ -f /run/.containerenv ] ; then
+ die "Adding the container image to a container runtime (${image_type}) not supported if running from a container (e.g. 'kas-container')"
+ fi
+ skopeo --insecure-policy copy \
+ "oci:${oci_img_dir}:${full_tag}" \
+ "${image_type}:${image_name}:latest"
+ ;;
+ *)
+ die "Unsupported format for containerize_rootfs: ${image_type}"
+ ;;
+ esac
+ done
+}
+
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index eddc444..ec93cab 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -76,6 +76,7 @@ inherit image-tools-extension
inherit image-postproc-extension
inherit image-locales-extension
inherit image-account-extension
+inherit image-container-extension
# Extra space for rootfs in MB
ROOTFS_EXTRA ?= "64"
--
2.30.1
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v6 2/5] classes: add new image type 'container-img'
2021-03-09 20:52 [PATCH v6 0/5] Add support for containerized root filesystems Silvano Cirujano Cuesta
2021-03-09 20:52 ` [PATCH v6 1/5] classes: add root filesystem containerizing class Silvano Cirujano Cuesta
@ 2021-03-09 20:52 ` Silvano Cirujano Cuesta
2021-03-09 20:52 ` [PATCH v6 3/5] sdk: add support for containerized sdk Silvano Cirujano Cuesta
` (4 subsequent siblings)
6 siblings, 0 replies; 11+ messages in thread
From: Silvano Cirujano Cuesta @ 2021-03-09 20:52 UTC (permalink / raw)
To: isar-users
Add a new "image" class for generating a container image containing the
target root filesystem.
Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>
---
meta/classes/container-img.bbclass | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
create mode 100644 meta/classes/container-img.bbclass
diff --git a/meta/classes/container-img.bbclass b/meta/classes/container-img.bbclass
new file mode 100644
index 0000000..79ef3e8
--- /dev/null
+++ b/meta/classes/container-img.bbclass
@@ -0,0 +1,18 @@
+# This software is a part of ISAR.
+# Copyright (C) Siemens AG, 2021
+#
+# SPDX-License-Identifier: MIT
+#
+# This class provides the task 'containerize_rootfs'
+# to create container images containing the target rootfs.
+
+do_container_image[stamp-extra-info] = "${DISTRO}-${MACHINE}"
+do_container_image[vardeps] += "CONTAINER_FORMATS"
+do_container_image(){
+ rootfs_id="${DISTRO}-${DISTRO_ARCH}"
+
+ bbdebug 1 "Generate container image in these formats: ${CONTAINER_FORMATS}"
+ containerize_rootfs "${IMAGE_ROOTFS}" "${rootfs_id}" "${CONTAINER_FORMATS}"
+}
+
+addtask container_image before do_image after do_image_tools
--
2.30.1
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v6 3/5] sdk: add support for containerized sdk
2021-03-09 20:52 [PATCH v6 0/5] Add support for containerized root filesystems Silvano Cirujano Cuesta
2021-03-09 20:52 ` [PATCH v6 1/5] classes: add root filesystem containerizing class Silvano Cirujano Cuesta
2021-03-09 20:52 ` [PATCH v6 2/5] classes: add new image type 'container-img' Silvano Cirujano Cuesta
@ 2021-03-09 20:52 ` Silvano Cirujano Cuesta
2021-03-09 20:52 ` [PATCH v6 4/5] docs: document creation of container images Silvano Cirujano Cuesta
` (3 subsequent siblings)
6 siblings, 0 replies; 11+ messages in thread
From: Silvano Cirujano Cuesta @ 2021-03-09 20:52 UTC (permalink / raw)
To: isar-users
Extend also task "populate_sdk" to support the creation of a container image
containing the SDK.
Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>
---
meta/classes/image-sdk-extension.bbclass | 42 ++++++++++++++++++++----
1 file changed, 35 insertions(+), 7 deletions(-)
diff --git a/meta/classes/image-sdk-extension.bbclass b/meta/classes/image-sdk-extension.bbclass
index a8c708a..426b925 100644
--- a/meta/classes/image-sdk-extension.bbclass
+++ b/meta/classes/image-sdk-extension.bbclass
@@ -6,11 +6,25 @@
# This class extends the image.bbclass to supply the creation of a sdk
SDK_INCLUDE_ISAR_APT ?= "0"
+SDK_FORMATS ?= "tar-xz"
+
+sdk_tar_xz() {
+ # Copy mount_chroot.sh for convenience
+ sudo cp ${SCRIPTSDIR}/mount_chroot.sh ${SDKCHROOT_DIR}
+
+ # Create SDK archive
+ cd -P ${SDKCHROOT_DIR}/..
+ sudo tar --transform="s|^rootfs|sdk-${DISTRO}-${DISTRO_ARCH}|" \
+ -c rootfs | xz -T0 > ${DEPLOY_DIR_IMAGE}/sdk-${DISTRO}-${DISTRO_ARCH}.tar.xz
+ bbdebug 1 "SDK rootfs available in ${DEPLOY_DIR_IMAGE}/sdk-${DISTRO}-${DISTRO_ARCH}.tar.xz"
+}
do_populate_sdk[stamp-extra-info] = "${DISTRO}-${MACHINE}"
do_populate_sdk[depends] = "sdkchroot:do_build"
-do_populate_sdk[vardeps] += "SDK_INCLUDE_ISAR_APT"
+do_populate_sdk[vardeps] += "SDK_INCLUDE_ISAR_APT SDK_FORMATS"
do_populate_sdk() {
+ local sdk_container_formats=""
+
if [ "${SDK_INCLUDE_ISAR_APT}" = "1" ]; then
# Copy isar-apt with deployed Isar packages
sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} ${SDKCHROOT_DIR}/isar-apt
@@ -48,12 +62,26 @@ do_populate_sdk() {
done
done
- # Copy mount_chroot.sh for convenience
- sudo cp ${SCRIPTSDIR}/mount_chroot.sh ${SDKCHROOT_DIR}
+ # separate SDK formats: TAR and container formats
+ for sdk_format in ${SDK_FORMATS} ; do
+ case ${sdk_format} in
+ "tar-xz")
+ sdk_tar_xz
+ ;;
+ "docker-archive" | "oci" | "oci-archive" | "docker-daemon" | "containers-storage")
+ sdk_container_formats="${sdk_container_formats} ${sdk_format}"
+ ;;
+ *)
+ die "unsupported SDK format specified: ${sdk_format}"
+ ;;
+ esac
+ done
- # Create SDK archive
- cd -P ${SDKCHROOT_DIR}/..
- sudo tar --transform="s|^rootfs|sdk-${DISTRO}-${DISTRO_ARCH}|" \
- -c rootfs | xz -T0 > ${DEPLOY_DIR_IMAGE}/sdk-${DISTRO}-${DISTRO_ARCH}.tar.xz
+ # generate the SDK in all the desired container formats
+ if [ -n "${sdk_container_formats}" ] ; then
+ bbnote "Generating SDK container in ${sdk_container_formats} format"
+ containerize_rootfs "${SDKCHROOT_DIR}" "sdk-${DISTRO}-${DISTRO_ARCH}" "${sdk_container_formats}"
+ fi
}
+
addtask populate_sdk after do_rootfs
--
2.30.1
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v6 4/5] docs: document creation of container images
2021-03-09 20:52 [PATCH v6 0/5] Add support for containerized root filesystems Silvano Cirujano Cuesta
` (2 preceding siblings ...)
2021-03-09 20:52 ` [PATCH v6 3/5] sdk: add support for containerized sdk Silvano Cirujano Cuesta
@ 2021-03-09 20:52 ` Silvano Cirujano Cuesta
2021-03-09 20:52 ` [PATCH v6 5/5] ci: add container image testing configurations Silvano Cirujano Cuesta
` (2 subsequent siblings)
6 siblings, 0 replies; 11+ messages in thread
From: Silvano Cirujano Cuesta @ 2021-03-09 20:52 UTC (permalink / raw)
To: isar-users
Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>
---
doc/user_manual.md | 127 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 127 insertions(+)
diff --git a/doc/user_manual.md b/doc/user_manual.md
index a4f3d1d..f6f49bc 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -19,6 +19,7 @@ Copyright (C) 2016-2019, ilbers GmbH
- [Add a Custom Application](#add-a-custom-application)
- [Enabling Cross-compilation](#isar-cross-compilation)
- [Create an ISAR SDK root filesystem](#create-an-isar-sdk-root-filesystem)
+ - [Create a containerized ISAR SDK root filesystem](#create-a-containerized-isar-sdk-root-filesystem)
- [Creation of local apt repo caching upstream Debian packages](#creation-of-local-apt-repo-caching-upstream-debian-packages)
@@ -84,6 +85,9 @@ If your host is >= buster, also install the following package.
apt install python3-distutils
```
+If you want to generate containerized SDKs, also install the following packages: `umoci` and `skopeo`.
+Umoci is provided by Debian Buster and can be installed with `apt install umoci`, Skopeo is provided by Debian Bullseye/Unstable and has to be installed either manually downloading the DEB and installing it (no other packages required) or with `apt install -t bullseye skopeo` (if unstable/bullseye included in `/etc/apt/sources.list[.d]`).
+
Notes:
* BitBake requires Python 3.4+.
@@ -223,6 +227,54 @@ qemu-system-x86_64 -m 256M -nographic -bios edk2/Build/OvmfX64/RELEASE_*/FV/OVMF
qemu-system-i386 -m 256M -nographic -hda tmp/deploy/images/qemui386/isar-image-base-debian-buster-qemui386.wic.img
```
+### Generate container image with root-filesystem
+
+A runnable container image is generated if you set IMAGE_TYPE to 'container-img'.
+Getting a container image can be the main purpose of an ISAR configuration, but not only.
+A container image created from an ISAR configuration meant for bare-metal or virtual machines can be helpfull to test certain applications which requirements (e.g. libraries) can be easily resolved in a containerized environment.
+
+Container images can be generated in different formats, selected with the variable `CONTAINER_FORMAT`. One or more (whitespace separated) of following options can be given:
+ - `docker-archive`: (default) an archive containing a Docker image that can be imported with [`docker import`](https://docs.docker.com/engine/reference/commandline/import/)
+ - `docker-daemon`: resulting container image is made available on the local Docker Daemon
+ - `containers-storage`: resulting container image is made available to tools using containers/storage back-end (e.g. Podman, CRIO, buildah,...)
+ - `oci-archive`: an archive containing an OCI image, mostly for archiving as seed for any of the above formats
+
+Following formats don't work if running `bitbake ...` (to build the image) from inside of a container (e.g. using `kas-container`): `docker-daemon` and `containers-storage`.
+It's technically possible, but requires making host resources (e.g. the Docker Daemon socket) accessible in the container.
+What can endanger the stability and security of the host.
+
+The resulting container image archives (only for `docker-archive` and `oci-archive`) are made available as `tmp/deploy/images/${MACHINE}/${DISTRO}-${DISTRO_ARCH}-${container_format}.tar.xz` (being `container_format` each one of the formats specified in `CONTAINER_FORMAT`).
+
+### Example
+
+ - Make the relevant environment variables available to the task
+
+For one-shot builds (use `local.conf` otherwise):
+
+```
+export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE IMAGE_TYPE CONTAINER_FORMAT"
+export IMAGE_TYPE="container-img"
+export CONTAINER_FORMAT="docker-archive"
+```
+
+ - Trigger creation of container image from root filesystem
+
+```
+bitbake mc:qemuarm-buster:isar-image-base
+```
+
+ - Load the container image into the Docker Daemon
+
+```
+xzcat build/tmp/deploy/images/qemuarm/debian-buster-armhf-docker-archive.tar.xz | docker load
+```
+
+ - Run a container using the container image (following commands starting with `#~:` are to be run in the container)
+
+```
+docker run --rm -ti --volume "$(pwd):/build" isar-buster-armhf:latest
+```
+
---
## Terms and Definitions
@@ -834,6 +886,81 @@ ii crossbuild-essential-armhf 12.3 all Inf
~#
```
+## Create a containerized ISAR SDK root filesystem
+
+### Motivation
+
+Distributing and using the SDK root filesystem created following the instructions in "[Create an ISAR SDK root filesystem](#create-an-isar-sdk-root-filesystem)" becomes easier using container images (at least for those using containers anyway)
+A "containerized" SDK adds to those advantages of a normal SDK root filesystem the comfort of container images.
+
+### Approach
+
+Create container image with SDK root filesystem with installed cross-toolchain for target architecture and ability to install already prebuilt target binary artifacts.
+Developer:
+ - runs a container based on the resulting container image mounting the source code to be built,
+ - develops applications for target platform on the container and
+ - leaves the container getting the results on the mounted directory.
+
+### Solution
+
+User specifies the variable `SDK_FORMAT` providing a space-separated list of SDK formats to generate.
+
+Supported formats are:
+ - `tar-xz`: (default) is the non-containerized format that results from following the instructions in "[Create an ISAR SDK root filesystem](#create-an-isar-sdk-root-filesystem)"
+ - `docker-archive`: an archive containing a Docker image that can be imported with [`docker import`](https://docs.docker.com/engine/reference/commandline/import/)
+ - `docker-daemon`: resulting container image is made available on the local Docker Daemon
+ - `containers-storage`: resulting container image is made available to tools using containers/storage back-end (e.g. Podman, CRIO, buildah,...)
+ - `oci-archive`: an archive containing an OCI image, mostly for archiving as seed for any of the above formats
+
+User manually triggers creation of SDK formats for his target platform by launching the task `do_populate_sdk` for target image, f.e.
+`bitbake -c do_populate_sdk mc:${MACHINE}-${DISTRO}:isar-image-base`.
+Packages that should be additionally installed into the SDK can be appended to `SDK_PREINSTALL` (external repositories) and `SDK_INSTALL` (self-built).
+
+Following formats don't work if running `bitbake -c do_populate_sdk ...` (to generate the containerized SDK) from inside of a container (e.g. using `kas-container`): `docker-daemon` and `containers-storage`.
+It's technically possible, but requires making host resources (e.g. the Docker Daemon socket) accessible in the container.
+What can endanger the stability and security of the host.
+
+The resulting SDK formats are archived into `tmp/deploy/images/${MACHINE}/sdk-${DISTRO}-${DISTRO_ARCH}-${sdk_format}.tar.xz` (being `sdk_format` each one of the formats specified in `SDK_FORMATS`).
+The SDK container directory `/isar-apt` contains a copy of isar-apt repo with locally prebuilt target debian packages (for <HOST_DISTRO>).
+One may get into an SDK container and install required target packages with the help of `apt-get install <package_name>:<DISTRO_ARCH>` command.
+The directory with the source code to develop on should be mounted on the container (with `--volume <host-directory>:<container-directory>`) to be able to edit files in the host with an IDE and build in the container.
+
+### Example
+
+ - Make the SDK formats to generate available to the task
+
+For one-shot builds (use `local.conf` otherwise):
+
+```
+export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE SDK_FORMATS"
+export SDK_FORMATS="docker-archive"
+```
+
+ - Trigger creation of SDK root filesystem
+
+```
+bitbake -c do_populate_sdk mc:qemuarm-buster:isar-image-base
+```
+
+ - Load the SDK container image into the Docker Daemon
+
+```
+xzcat build/tmp/deploy/images/qemuarm/sdk-debian-buster-armhf-docker-archive.tar.xz | docker load
+```
+
+ - Run a container using the SDK container image (following commands starting with `#~:` are to be run in the container)
+
+```
+docker run --rm -ti --volume "$(pwd):/build" isar-sdk-buster-armhf:latest
+```
+
+ - Check that cross toolchains are installed
+
+```
+:~# dpkg -l | grep crossbuild-essential-armhf
+ii crossbuild-essential-armhf 12.3 all Informational list of cross-build-essential packages
+```
+
## Creation of local apt repo caching upstream Debian packages
### Motivation
--
2.30.1
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v6 5/5] ci: add container image testing configurations
2021-03-09 20:52 [PATCH v6 0/5] Add support for containerized root filesystems Silvano Cirujano Cuesta
` (3 preceding siblings ...)
2021-03-09 20:52 ` [PATCH v6 4/5] docs: document creation of container images Silvano Cirujano Cuesta
@ 2021-03-09 20:52 ` Silvano Cirujano Cuesta
2021-03-10 16:18 ` [PATCH v6 0/5] Add support for containerized root filesystems Anton Mikanovich
2021-03-10 16:28 ` Henning Schild
6 siblings, 0 replies; 11+ messages in thread
From: Silvano Cirujano Cuesta @ 2021-03-09 20:52 UTC (permalink / raw)
To: isar-users
Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>
---
meta-isar/conf/machine/container.conf | 5 +++++
meta-isar/conf/multiconfig/container-bullseye.conf | 4 ++++
meta-isar/conf/multiconfig/container-buster.conf | 4 ++++
meta-isar/conf/multiconfig/container-focal.conf | 4 ++++
meta-isar/conf/multiconfig/container-stretch.conf | 4 ++++
scripts/ci_build.sh | 11 ++++++++++-
6 files changed, 31 insertions(+), 1 deletion(-)
create mode 100644 meta-isar/conf/machine/container.conf
create mode 100644 meta-isar/conf/multiconfig/container-bullseye.conf
create mode 100644 meta-isar/conf/multiconfig/container-buster.conf
create mode 100644 meta-isar/conf/multiconfig/container-focal.conf
create mode 100644 meta-isar/conf/multiconfig/container-stretch.conf
diff --git a/meta-isar/conf/machine/container.conf b/meta-isar/conf/machine/container.conf
new file mode 100644
index 0000000..367d790
--- /dev/null
+++ b/meta-isar/conf/machine/container.conf
@@ -0,0 +1,5 @@
+# This software is a part of ISAR.
+# Copyright (C) 2015-2017 ilbers GmbH
+
+IMAGE_TYPE = "container-img"
+
diff --git a/meta-isar/conf/multiconfig/container-bullseye.conf b/meta-isar/conf/multiconfig/container-bullseye.conf
new file mode 100644
index 0000000..200b241
--- /dev/null
+++ b/meta-isar/conf/multiconfig/container-bullseye.conf
@@ -0,0 +1,4 @@
+# This software is a part of ISAR.
+
+MACHINE ?= "container"
+DISTRO ?= "debian-bullseye"
diff --git a/meta-isar/conf/multiconfig/container-buster.conf b/meta-isar/conf/multiconfig/container-buster.conf
new file mode 100644
index 0000000..78b0324
--- /dev/null
+++ b/meta-isar/conf/multiconfig/container-buster.conf
@@ -0,0 +1,4 @@
+# This software is a part of ISAR.
+
+MACHINE ?= "container"
+DISTRO ?= "debian-buster"
diff --git a/meta-isar/conf/multiconfig/container-focal.conf b/meta-isar/conf/multiconfig/container-focal.conf
new file mode 100644
index 0000000..10e0071
--- /dev/null
+++ b/meta-isar/conf/multiconfig/container-focal.conf
@@ -0,0 +1,4 @@
+# This software is a part of ISAR.
+
+MACHINE ?= "container"
+DISTRO ?= "ubuntu-focal"
diff --git a/meta-isar/conf/multiconfig/container-stretch.conf b/meta-isar/conf/multiconfig/container-stretch.conf
new file mode 100644
index 0000000..3ff8bcb
--- /dev/null
+++ b/meta-isar/conf/multiconfig/container-stretch.conf
@@ -0,0 +1,4 @@
+# This software is a part of ISAR.
+
+MACHINE ?= "container"
+DISTRO ?= "debian-stretch"
diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 3868fb6..c3d49dd 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -33,7 +33,11 @@ TARGETS_SET="\
mc:qemumipsel-buster:isar-image-base \
mc:nand-ubi-demo-buster:isar-image-ubi \
mc:rpi-stretch:isar-image-base \
- mc:qemuamd64-focal:isar-image-base"
+ mc:qemuamd64-focal:isar-image-base \
+ mc:container-stretch:isar-image-base \
+ mc:container-buster:isar-image-base \
+ mc:container-focal:isar-image-base \
+ "
# qemu-user-static of <= buster too old to build that
# mc:qemuarm64-buster:isar-image-base
# mc:qemuarm64-bullseye:isar-image-base
@@ -43,6 +47,7 @@ TARGETS_SET_BULLSEYE="\
mc:qemuarm-bullseye:isar-image-base \
mc:qemui386-bullseye:isar-image-base \
mc:qemumipsel-bullseye:isar-image-base \
+ mc:container-bullseye:isar-image-base \
"
CROSS_TARGETS_SET="\
@@ -201,6 +206,10 @@ fi
# In addition test SDK creation
bitbake $BB_ARGS -c do_populate_sdk mc:qemuarm-stretch:isar-image-base
while [ -e bitbake.sock ]; do sleep 1; done
+# and SDK container image creation
+SDK_FORMAT="docker-archive" BB_ENV_WHITELIST="$BB_ENV_WHITELIST SDK_FORMAT" \
+ bitbake $BB_ARGS -c do_populate_sdk mc:qemuarm-stretch:isar-image-base
+while [ -e bitbake.sock ]; do sleep 1; done
if [ -z "$FAST_BUILD" ]; then
# Cleanup and disable cross build
--
2.30.1
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v6 0/5] Add support for containerized root filesystems
2021-03-09 20:52 [PATCH v6 0/5] Add support for containerized root filesystems Silvano Cirujano Cuesta
` (4 preceding siblings ...)
2021-03-09 20:52 ` [PATCH v6 5/5] ci: add container image testing configurations Silvano Cirujano Cuesta
@ 2021-03-10 16:18 ` Anton Mikanovich
2021-03-10 17:18 ` Silvano Cirujano Cuesta
2021-03-10 16:28 ` Henning Schild
6 siblings, 1 reply; 11+ messages in thread
From: Anton Mikanovich @ 2021-03-10 16:18 UTC (permalink / raw)
To: Silvano Cirujano Cuesta, isar-users
09.03.2021 23:52, Silvano Cirujano Cuesta wrote:
> This patch series provides support for containerized root filesystems,
> for both target images and SDKs.
>
> For containerized target images the new image type `container-img` has
> been added.
>
> For containerized SDKs the task `populate_sdk` has been extended.
>
> Containerized root filesystems are easy to distribute and run, enabling
> this way following scenarios:
> - Use ISAR to build container images meant to be run only in containers.
> - Use the same ISAR configuration to build images for containers, VMs
> and bare-metal.
> - Easy SDK distribution and "installation".
> - Quickly testing certain applications in the workstation using the
> target root filesystem.
>
> In order to build containerized target root filesystems `IMAGE_TYPE` has
> to be `container-img`, additionally the container image format can be
> selected with the variable `CONTAINER_FORMATS`. The default format is
> `docker-archive`.
>
> In order to build containerized SDKs the variable `SDK_FORMAT` has to
> provide any of the supported container formats (e.g. `docker-archive`).
> The default format is the legacy non-containerized: `tar_xz`.
>
> It also provides a sample machine, multiconfigs and ci-testing.
>
> More information about its usage is documented in the file
> docs/user_manual.md.
>
> A PoC/demo of this functionality (only the SDK part) has been created
> based on the project https://github.com/siemens/meta-iot2050.
> Jan Kiszka already tested and liked it! =>
> https://github.com/siemens/meta-iot2050/issues/86#issuecomment-768907845
>
> Successful builds of both containerized target and SDK are available on
> the same PoC project:
> - https://github.com/Silvanoc/meta-iot2050/actions/runs/558311580
> - https://github.com/Silvanoc/meta-iot2050/actions/runs/558311581
> and also the resulting images:
> - https://github.com/users/Silvanoc/packages/container/package/meta-iot2050%2Fiot2050-debian-arm64
> - https://github.com/users/Silvanoc/packages/container/package/meta-iot2050%2Fiot2050-debian-sdk-arm64
>
> In order to get a feeling about its usage (you need Docker or Podman),
> follow these simple copy&paste instructions:
> https://github.com/Silvanoc/meta-iot2050/blob/master/kas/BUILDING-SDK-CONTAINER.md#running-the-sdk
> Build instructions are available in the upper part of that document.
>
> Two new dependencies (umoci and skopeo -backporting from bullseye to
> buster works easily) are required to create containerized root
> filesystems (as specified in the documentation).
>
> Typical container image management actions (e.g. push an image to a
> container image regitry) are out of scope. Available tools (Docker,
> Skopeo, Buildah, Podman,...) should be used for these actions.
>
> A patch will follow this one to get the dependencies into the container
> images being provided by the project
> https://github.com/siemens/kas (for `kas-container`, for example).
>
> Silvano Cirujano Cuesta (5):
> classes: add root filesystem containerizing class
> classes: add new image type 'container-img'
> sdk: add support for containerized sdk
> docs: document creation of container images
> ci: add container image testing configurations
>
> doc/user_manual.md | 127 ++++++++++++++++++
> meta-isar/conf/machine/container.conf | 5 +
> .../conf/multiconfig/container-bullseye.conf | 4 +
> .../conf/multiconfig/container-buster.conf | 4 +
> .../conf/multiconfig/container-focal.conf | 4 +
> .../conf/multiconfig/container-stretch.conf | 4 +
> meta/classes/container-img.bbclass | 18 +++
> .../classes/image-container-extension.bbclass | 81 +++++++++++
> meta/classes/image-sdk-extension.bbclass | 42 +++++-
> meta/classes/image.bbclass | 1 +
> scripts/ci_build.sh | 11 +-
> 11 files changed, 293 insertions(+), 8 deletions(-)
> create mode 100644 meta-isar/conf/machine/container.conf
> create mode 100644 meta-isar/conf/multiconfig/container-bullseye.conf
> create mode 100644 meta-isar/conf/multiconfig/container-buster.conf
> create mode 100644 meta-isar/conf/multiconfig/container-focal.conf
> create mode 100644 meta-isar/conf/multiconfig/container-stretch.conf
> create mode 100644 meta/classes/container-img.bbclass
> create mode 100644 meta/classes/image-container-extension.bbclass
>
Thanks for adding CI cases, but Jenkins failed during do_populate_sdk:
14:17:17 + SDK_FORMAT=docker-archive
14:17:17 + BB_ENV_WHITELIST=' SDK_FORMAT'
14:17:17 + bitbake -c do_populate_sdk mc:qemuarm-stretch:isar-image-base
...
14:17:18 ERROR: Unable to parse
/workspace/build/isar_am_devel_fast/92/meta-isar/recipes-app/libhello/libhello.bb
14:17:18 Traceback (most recent call last):
14:17:18 File
"/workspace/build/isar_am_devel_fast/92/bitbake/lib/bb/parse/ast.py",
line 382, in
multi_finalize(fn='/workspace/build/isar_am_devel_fast/92/meta-isar/recipes-app/libhello/libhello.bb',
d=<bb.data_smart.DataSmart object at 0x7f12917e1c50>):
14:17:18 try:
14:17:18 > finalize(fn, d)
14:17:18 except bb.parse.SkipRecipe as e:
14:17:18 File
"/workspace/build/isar_am_devel_fast/92/bitbake/lib/bb/parse/ast.py",
line 347, in
finalize(fn='/workspace/build/isar_am_devel_fast/92/meta-isar/recipes-app/libhello/libhello.bb',
d=<bb.data_smart.DataSmart object at 0x7f12917e1c50>, variant=None):
14:17:18
14:17:18 > bb.parse.siggen.finalise(fn, d, variant)
14:17:18
14:17:18 File
"/workspace/build/isar_am_devel_fast/92/bitbake/lib/bb/siggen.py", line
182, in
SignatureGeneratorBasicHash.finalise(fn='/workspace/build/isar_am_devel_fast/92/meta-isar/recipes-app/libhello/libhello.bb',
d=<bb.data_smart.DataSmart object at 0x7f12917e1c50>, variant=None):
14:17:18 try:
14:17:18 > taskdeps = self._build_data(fn, d)
14:17:18 except bb.parse.SkipRecipe:
14:17:18 File
"/workspace/build/isar_am_devel_fast/92/bitbake/lib/bb/siggen.py", line
150, in
SignatureGeneratorBasicHash._build_data(fn='/workspace/build/isar_am_devel_fast/92/meta-isar/recipes-app/libhello/libhello.bb',
d=<bb.data_smart.DataSmart object at 0x7f12917e1c50>):
14:17:18 ignore_mismatch =
((d.getVar("BB_HASH_IGNORE_MISMATCH") or '') == '1')
14:17:18 > tasklist, gendeps, lookupcache =
bb.data.generate_dependencies(d, self.basewhitelist)
14:17:18
14:17:18 File
"/workspace/build/isar_am_devel_fast/92/bitbake/lib/bb/data.py", line
371, in generate_dependencies(d=<bb.data_smart.DataSmart object at
0x7f12917e1c50>, whitelist={'SHELL', 'DEPLOY_DIR', 'PRSERV_DUMPFILE',
'SDKPKGSUFFIX', 'FILE_DIRNAME', 'TERM', 'COREBASE', 'LICENSE_PATH',
'STAGING_DIR_HOST', 'SSTATE_PKGARCH', 'PKGDATA_DIR', 'THISDIR', 'USER',
'EXTERNAL_TOOLCHAIN', 'PWD', 'DL_DIR', 'LOGNAME', 'PATH',
'BB_LIMITEDDEPS', 'CCACHE', 'BB_TASKHASH', 'CCACHE_DIR',
'PARALLEL_MAKE', 'BBSERVER', 'PRSERV_LOCKDOWN', 'BBPATH',
'BB_WORKERCONTEXT', 'PRSERV_HOST', 'WORKDIR', 'FILE',
'CCACHE_NOHASHDIR', 'PRSERV_DUMPDIR', 'FILESEXTRAPATHS', 'HOME',
'BUILD_ARCH', 'STAMPCLEAN', 'SSTATE_DIR', 'TMPDIR', 'FILESPATH',
'STAGING_DIR_TARGET'}):
14:17:18 keys = set(key for key in d if not key.startswith("__"))
14:17:18 > shelldeps = set(key for key in
d.getVar("__exportlist", False) if d.getVarFlag(key, "export", False)
and not d.getVarFlag(key, "unexport", False))
14:17:18 varflagsexcl = d.getVar('BB_SIGNATURE_EXCLUDE_FLAGS')
14:17:18 TypeError: 'NoneType' object is not iterable
Full log:
http://ci.isar-build.org:8080/job/isar_am_devel_fast/92/consoleFull
--
Anton Mikanovich
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v6 0/5] Add support for containerized root filesystems
2021-03-09 20:52 [PATCH v6 0/5] Add support for containerized root filesystems Silvano Cirujano Cuesta
` (5 preceding siblings ...)
2021-03-10 16:18 ` [PATCH v6 0/5] Add support for containerized root filesystems Anton Mikanovich
@ 2021-03-10 16:28 ` Henning Schild
6 siblings, 0 replies; 11+ messages in thread
From: Henning Schild @ 2021-03-10 16:28 UTC (permalink / raw)
To: [ext] Silvano Cirujano Cuesta; +Cc: isar-users
Silvano you might want to push this to
https://code.siemens.com/ebsy/debian/isar to make use of our CI.
Henning
Am Tue, 9 Mar 2021 21:52:34 +0100
schrieb "[ext] Silvano Cirujano Cuesta"
<silvano.cirujano-cuesta@siemens.com>:
> This patch series provides support for containerized root filesystems,
> for both target images and SDKs.
>
> For containerized target images the new image type `container-img` has
> been added.
>
> For containerized SDKs the task `populate_sdk` has been extended.
>
> Containerized root filesystems are easy to distribute and run,
> enabling this way following scenarios:
> - Use ISAR to build container images meant to be run only in
> containers.
> - Use the same ISAR configuration to build images for containers, VMs
> and bare-metal.
> - Easy SDK distribution and "installation".
> - Quickly testing certain applications in the workstation using the
> target root filesystem.
>
> In order to build containerized target root filesystems `IMAGE_TYPE`
> has to be `container-img`, additionally the container image format
> can be selected with the variable `CONTAINER_FORMATS`. The default
> format is `docker-archive`.
>
> In order to build containerized SDKs the variable `SDK_FORMAT` has to
> provide any of the supported container formats (e.g.
> `docker-archive`). The default format is the legacy
> non-containerized: `tar_xz`.
>
> It also provides a sample machine, multiconfigs and ci-testing.
>
> More information about its usage is documented in the file
> docs/user_manual.md.
>
> A PoC/demo of this functionality (only the SDK part) has been created
> based on the project https://github.com/siemens/meta-iot2050.
> Jan Kiszka already tested and liked it! =>
> https://github.com/siemens/meta-iot2050/issues/86#issuecomment-768907845
>
> Successful builds of both containerized target and SDK are available
> on the same PoC project:
> - https://github.com/Silvanoc/meta-iot2050/actions/runs/558311580
> - https://github.com/Silvanoc/meta-iot2050/actions/runs/558311581
> and also the resulting images:
> -
> https://github.com/users/Silvanoc/packages/container/package/meta-iot2050%2Fiot2050-debian-arm64
> -
> https://github.com/users/Silvanoc/packages/container/package/meta-iot2050%2Fiot2050-debian-sdk-arm64
>
> In order to get a feeling about its usage (you need Docker or Podman),
> follow these simple copy&paste instructions:
> https://github.com/Silvanoc/meta-iot2050/blob/master/kas/BUILDING-SDK-CONTAINER.md#running-the-sdk
> Build instructions are available in the upper part of that document.
>
> Two new dependencies (umoci and skopeo -backporting from bullseye to
> buster works easily) are required to create containerized root
> filesystems (as specified in the documentation).
>
> Typical container image management actions (e.g. push an image to a
> container image regitry) are out of scope. Available tools (Docker,
> Skopeo, Buildah, Podman,...) should be used for these actions.
>
> A patch will follow this one to get the dependencies into the
> container images being provided by the project
> https://github.com/siemens/kas (for `kas-container`, for example).
>
> Silvano Cirujano Cuesta (5):
> classes: add root filesystem containerizing class
> classes: add new image type 'container-img'
> sdk: add support for containerized sdk
> docs: document creation of container images
> ci: add container image testing configurations
>
> doc/user_manual.md | 127
> ++++++++++++++++++ meta-isar/conf/machine/container.conf |
> 5 + .../conf/multiconfig/container-bullseye.conf | 4 +
> .../conf/multiconfig/container-buster.conf | 4 +
> .../conf/multiconfig/container-focal.conf | 4 +
> .../conf/multiconfig/container-stretch.conf | 4 +
> meta/classes/container-img.bbclass | 18 +++
> .../classes/image-container-extension.bbclass | 81 +++++++++++
> meta/classes/image-sdk-extension.bbclass | 42 +++++-
> meta/classes/image.bbclass | 1 +
> scripts/ci_build.sh | 11 +-
> 11 files changed, 293 insertions(+), 8 deletions(-)
> create mode 100644 meta-isar/conf/machine/container.conf
> create mode 100644 meta-isar/conf/multiconfig/container-bullseye.conf
> create mode 100644 meta-isar/conf/multiconfig/container-buster.conf
> create mode 100644 meta-isar/conf/multiconfig/container-focal.conf
> create mode 100644 meta-isar/conf/multiconfig/container-stretch.conf
> create mode 100644 meta/classes/container-img.bbclass
> create mode 100644 meta/classes/image-container-extension.bbclass
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v6 0/5] Add support for containerized root filesystems
2021-03-10 16:18 ` [PATCH v6 0/5] Add support for containerized root filesystems Anton Mikanovich
@ 2021-03-10 17:18 ` Silvano Cirujano Cuesta
2021-03-10 17:25 ` Henning Schild
0 siblings, 1 reply; 11+ messages in thread
From: Silvano Cirujano Cuesta @ 2021-03-10 17:18 UTC (permalink / raw)
To: Anton Mikanovich, isar-users
On 10/03/2021 17:18, Anton Mikanovich wrote:
> 09.03.2021 23:52, Silvano Cirujano Cuesta wrote:
>> This patch series provides support for containerized root filesystems,
>> for both target images and SDKs.
>>
>> For containerized target images the new image type `container-img` has
>> been added.
>>
>> For containerized SDKs the task `populate_sdk` has been extended.
>>
>> Containerized root filesystems are easy to distribute and run, enabling
>> this way following scenarios:
>> - Use ISAR to build container images meant to be run only in containers.
>> - Use the same ISAR configuration to build images for containers, VMs
>> and bare-metal.
>> - Easy SDK distribution and "installation".
>> - Quickly testing certain applications in the workstation using the
>> target root filesystem.
>>
>> In order to build containerized target root filesystems `IMAGE_TYPE` has
>> to be `container-img`, additionally the container image format can be
>> selected with the variable `CONTAINER_FORMATS`. The default format is
>> `docker-archive`.
>>
>> In order to build containerized SDKs the variable `SDK_FORMAT` has to
>> provide any of the supported container formats (e.g. `docker-archive`).
>> The default format is the legacy non-containerized: `tar_xz`.
>>
>> It also provides a sample machine, multiconfigs and ci-testing.
>>
>> More information about its usage is documented in the file
>> docs/user_manual.md.
>>
>> A PoC/demo of this functionality (only the SDK part) has been created
>> based on the project https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsiemens%2Fmeta-iot2050&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C70ab5dc719fe445169a008d8e3e01c27%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637509898980089588%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2FB%2BNBKhRCkVNy%2FGEbAiFgEUZBpy0VmhkWdpvNq2wokM%3D&reserved=0.
>> Jan Kiszka already tested and liked it! =>
>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsiemens%2Fmeta-iot2050%2Fissues%2F86%23issuecomment-768907845&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C70ab5dc719fe445169a008d8e3e01c27%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637509898980089588%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Os3cY15D2myT5sdPINARBh63iwb3GpE5IlupRGnA0v0%3D&reserved=0
>>
>> Successful builds of both containerized target and SDK are available on
>> the same PoC project:
>> - https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FSilvanoc%2Fmeta-iot2050%2Factions%2Fruns%2F558311580&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C70ab5dc719fe445169a008d8e3e01c27%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637509898980089588%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=N3DUdGxhNOWlvbatGxopSWPWsMXsXdvsKmIsOh1KtyU%3D&reserved=0
>> - https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FSilvanoc%2Fmeta-iot2050%2Factions%2Fruns%2F558311581&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C70ab5dc719fe445169a008d8e3e01c27%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637509898980089588%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=cOkongJ%2BtbrEvaquB241NbaK1nQHHiY%2FrgbYHPFAjeo%3D&reserved=0
>> and also the resulting images:
>> - https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fusers%2FSilvanoc%2Fpackages%2Fcontainer%2Fpackage%2Fmeta-iot2050%252Fiot2050-debian-arm64&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C70ab5dc719fe445169a008d8e3e01c27%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637509898980089588%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=UegaXdy58CUxX8%2FsDnG8cy3V9uy9IUyh8yA00U%2FSaR4%3D&reserved=0
>> - https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fusers%2FSilvanoc%2Fpackages%2Fcontainer%2Fpackage%2Fmeta-iot2050%252Fiot2050-debian-sdk-arm64&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C70ab5dc719fe445169a008d8e3e01c27%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637509898980089588%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=wss3OIVsS2ulaQ4Jr7ria0wMna6Y0U8Ahp9M3XQU5PM%3D&reserved=0
>>
>> In order to get a feeling about its usage (you need Docker or Podman),
>> follow these simple copy&paste instructions:
>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FSilvanoc%2Fmeta-iot2050%2Fblob%2Fmaster%2Fkas%2FBUILDING-SDK-CONTAINER.md%23running-the-sdk&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C70ab5dc719fe445169a008d8e3e01c27%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637509898980089588%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2FiIv247NEqKV%2F2osF6hL5S4QiA6T8yqd5TQpi3EbuGQ%3D&reserved=0
>> Build instructions are available in the upper part of that document.
>>
>> Two new dependencies (umoci and skopeo -backporting from bullseye to
>> buster works easily) are required to create containerized root
>> filesystems (as specified in the documentation).
>>
>> Typical container image management actions (e.g. push an image to a
>> container image regitry) are out of scope. Available tools (Docker,
>> Skopeo, Buildah, Podman,...) should be used for these actions.
>>
>> A patch will follow this one to get the dependencies into the container
>> images being provided by the project
>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsiemens%2Fkas&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C70ab5dc719fe445169a008d8e3e01c27%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637509898980099579%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=eozG3LQR9pRGIcNiz9a0DckXhkBu8pcas9c%2F%2BWon9mU%3D&reserved=0 (for `kas-container`, for example).
>>
>> Silvano Cirujano Cuesta (5):
>> classes: add root filesystem containerizing class
>> classes: add new image type 'container-img'
>> sdk: add support for containerized sdk
>> docs: document creation of container images
>> ci: add container image testing configurations
>>
>> doc/user_manual.md | 127 ++++++++++++++++++
>> meta-isar/conf/machine/container.conf | 5 +
>> .../conf/multiconfig/container-bullseye.conf | 4 +
>> .../conf/multiconfig/container-buster.conf | 4 +
>> .../conf/multiconfig/container-focal.conf | 4 +
>> .../conf/multiconfig/container-stretch.conf | 4 +
>> meta/classes/container-img.bbclass | 18 +++
>> .../classes/image-container-extension.bbclass | 81 +++++++++++
>> meta/classes/image-sdk-extension.bbclass | 42 +++++-
>> meta/classes/image.bbclass | 1 +
>> scripts/ci_build.sh | 11 +-
>> 11 files changed, 293 insertions(+), 8 deletions(-)
>> create mode 100644 meta-isar/conf/machine/container.conf
>> create mode 100644 meta-isar/conf/multiconfig/container-bullseye.conf
>> create mode 100644 meta-isar/conf/multiconfig/container-buster.conf
>> create mode 100644 meta-isar/conf/multiconfig/container-focal.conf
>> create mode 100644 meta-isar/conf/multiconfig/container-stretch.conf
>> create mode 100644 meta/classes/container-img.bbclass
>> create mode 100644 meta/classes/image-container-extension.bbclass
>>
> Thanks for adding CI cases, but Jenkins failed during do_populate_sdk:
Do you have the documented dependencies (umoci and skopeo backported from bullseye) in the CI system? I've tested the commands manually on my system using kas-container and they worked, but your setup is slightly different.
>
> 14:17:17 + SDK_FORMAT=docker-archive
> 14:17:17 + BB_ENV_WHITELIST=' SDK_FORMAT'
> 14:17:17 + bitbake -c do_populate_sdk mc:qemuarm-stretch:isar-image-base
> ...
> 14:17:18 ERROR: Unable to parse /workspace/build/isar_am_devel_fast/92/meta-isar/recipes-app/libhello/libhello.bb
> 14:17:18 Traceback (most recent call last):
> 14:17:18 File "/workspace/build/isar_am_devel_fast/92/bitbake/lib/bb/parse/ast.py", line 382, in multi_finalize(fn='/workspace/build/isar_am_devel_fast/92/meta-isar/recipes-app/libhello/libhello.bb', d=<bb.data_smart.DataSmart object at 0x7f12917e1c50>):
> 14:17:18 try:
> 14:17:18 > finalize(fn, d)
> 14:17:18 except bb.parse.SkipRecipe as e:
> 14:17:18 File "/workspace/build/isar_am_devel_fast/92/bitbake/lib/bb/parse/ast.py", line 347, in finalize(fn='/workspace/build/isar_am_devel_fast/92/meta-isar/recipes-app/libhello/libhello.bb', d=<bb.data_smart.DataSmart object at 0x7f12917e1c50>, variant=None):
> 14:17:18
> 14:17:18 > bb.parse.siggen.finalise(fn, d, variant)
> 14:17:18
> 14:17:18 File "/workspace/build/isar_am_devel_fast/92/bitbake/lib/bb/siggen.py", line 182, in SignatureGeneratorBasicHash.finalise(fn='/workspace/build/isar_am_devel_fast/92/meta-isar/recipes-app/libhello/libhello.bb', d=<bb.data_smart.DataSmart object at 0x7f12917e1c50>, variant=None):
> 14:17:18 try:
> 14:17:18 > taskdeps = self._build_data(fn, d)
> 14:17:18 except bb.parse.SkipRecipe:
> 14:17:18 File "/workspace/build/isar_am_devel_fast/92/bitbake/lib/bb/siggen.py", line 150, in SignatureGeneratorBasicHash._build_data(fn='/workspace/build/isar_am_devel_fast/92/meta-isar/recipes-app/libhello/libhello.bb', d=<bb.data_smart.DataSmart object at 0x7f12917e1c50>):
> 14:17:18 ignore_mismatch = ((d.getVar("BB_HASH_IGNORE_MISMATCH") or '') == '1')
> 14:17:18 > tasklist, gendeps, lookupcache = bb.data.generate_dependencies(d, self.basewhitelist)
> 14:17:18
> 14:17:18 File "/workspace/build/isar_am_devel_fast/92/bitbake/lib/bb/data.py", line 371, in generate_dependencies(d=<bb.data_smart.DataSmart object at 0x7f12917e1c50>, whitelist={'SHELL', 'DEPLOY_DIR', 'PRSERV_DUMPFILE', 'SDKPKGSUFFIX', 'FILE_DIRNAME', 'TERM', 'COREBASE', 'LICENSE_PATH', 'STAGING_DIR_HOST', 'SSTATE_PKGARCH', 'PKGDATA_DIR', 'THISDIR', 'USER', 'EXTERNAL_TOOLCHAIN', 'PWD', 'DL_DIR', 'LOGNAME', 'PATH', 'BB_LIMITEDDEPS', 'CCACHE', 'BB_TASKHASH', 'CCACHE_DIR', 'PARALLEL_MAKE', 'BBSERVER', 'PRSERV_LOCKDOWN', 'BBPATH', 'BB_WORKERCONTEXT', 'PRSERV_HOST', 'WORKDIR', 'FILE', 'CCACHE_NOHASHDIR', 'PRSERV_DUMPDIR', 'FILESEXTRAPATHS', 'HOME', 'BUILD_ARCH', 'STAMPCLEAN', 'SSTATE_DIR', 'TMPDIR', 'FILESPATH', 'STAGING_DIR_TARGET'}):
> 14:17:18 keys = set(key for key in d if not key.startswith("__"))
> 14:17:18 > shelldeps = set(key for key in d.getVar("__exportlist", False) if d.getVarFlag(key, "export", False) and not d.getVarFlag(key, "unexport", False))
> 14:17:18 varflagsexcl = d.getVar('BB_SIGNATURE_EXCLUDE_FLAGS')
> 14:17:18 TypeError: 'NoneType' object is not iterable
>
> Full log: https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.isar-build.org%3A8080%2Fjob%2Fisar_am_devel_fast%2F92%2FconsoleFull&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C70ab5dc719fe445169a008d8e3e01c27%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637509898980099579%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=SS%2FYypaMBwrC9dvwJCeIrW9EjPqVfF2RXFghcfAn4Lc%3D&reserved=0
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v6 0/5] Add support for containerized root filesystems
2021-03-10 17:18 ` Silvano Cirujano Cuesta
@ 2021-03-10 17:25 ` Henning Schild
2021-03-10 17:30 ` Silvano Cirujano Cuesta
0 siblings, 1 reply; 11+ messages in thread
From: Henning Schild @ 2021-03-10 17:25 UTC (permalink / raw)
To: [ext] Silvano Cirujano Cuesta; +Cc: Anton Mikanovich, isar-users
Am Wed, 10 Mar 2021 18:18:23 +0100
schrieb "[ext] Silvano Cirujano Cuesta"
<silvano.cirujano-cuesta@siemens.com>:
> On 10/03/2021 17:18, Anton Mikanovich wrote:
> > 09.03.2021 23:52, Silvano Cirujano Cuesta wrote:
> >> This patch series provides support for containerized root
> >> filesystems, for both target images and SDKs.
> >>
> >> For containerized target images the new image type `container-img`
> >> has been added.
> >>
> >> For containerized SDKs the task `populate_sdk` has been extended.
> >>
> >> Containerized root filesystems are easy to distribute and run,
> >> enabling this way following scenarios:
> >> - Use ISAR to build container images meant to be run only in
> >> containers.
> >> - Use the same ISAR configuration to build images for
> >> containers, VMs and bare-metal.
> >> - Easy SDK distribution and "installation".
> >> - Quickly testing certain applications in the workstation using
> >> the target root filesystem.
> >>
> >> In order to build containerized target root filesystems
> >> `IMAGE_TYPE` has to be `container-img`, additionally the container
> >> image format can be selected with the variable
> >> `CONTAINER_FORMATS`. The default format is `docker-archive`.
> >>
> >> In order to build containerized SDKs the variable `SDK_FORMAT` has
> >> to provide any of the supported container formats (e.g.
> >> `docker-archive`). The default format is the legacy
> >> non-containerized: `tar_xz`.
> >>
> >> It also provides a sample machine, multiconfigs and ci-testing.
> >>
> >> More information about its usage is documented in the file
> >> docs/user_manual.md.
> >>
> >> A PoC/demo of this functionality (only the SDK part) has been
> >> created based on the project
> >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsiemens%2Fmeta-iot2050&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C70ab5dc719fe445169a008d8e3e01c27%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637509898980089588%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2FB%2BNBKhRCkVNy%2FGEbAiFgEUZBpy0VmhkWdpvNq2wokM%3D&reserved=0.
> >> Jan Kiszka already tested and liked it! =>
> >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsiemens%2Fmeta-iot2050%2Fissues%2F86%23issuecomment-768907845&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C70ab5dc719fe445169a008d8e3e01c27%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637509898980089588%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Os3cY15D2myT5sdPINARBh63iwb3GpE5IlupRGnA0v0%3D&reserved=0
> >>
> >> Successful builds of both containerized target and SDK are
> >> available on the same PoC project:
> >> -
> >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FSilvanoc%2Fmeta-iot2050%2Factions%2Fruns%2F558311580&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C70ab5dc719fe445169a008d8e3e01c27%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637509898980089588%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=N3DUdGxhNOWlvbatGxopSWPWsMXsXdvsKmIsOh1KtyU%3D&reserved=0
> >> -
> >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FSilvanoc%2Fmeta-iot2050%2Factions%2Fruns%2F558311581&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C70ab5dc719fe445169a008d8e3e01c27%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637509898980089588%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=cOkongJ%2BtbrEvaquB241NbaK1nQHHiY%2FrgbYHPFAjeo%3D&reserved=0
> >> and also the resulting images:
> >> -
> >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fusers%2FSilvanoc%2Fpackages%2Fcontainer%2Fpackage%2Fmeta-iot2050%252Fiot2050-debian-arm64&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C70ab5dc719fe445169a008d8e3e01c27%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637509898980089588%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=UegaXdy58CUxX8%2FsDnG8cy3V9uy9IUyh8yA00U%2FSaR4%3D&reserved=0
> >> -
> >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fusers%2FSilvanoc%2Fpackages%2Fcontainer%2Fpackage%2Fmeta-iot2050%252Fiot2050-debian-sdk-arm64&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C70ab5dc719fe445169a008d8e3e01c27%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637509898980089588%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=wss3OIVsS2ulaQ4Jr7ria0wMna6Y0U8Ahp9M3XQU5PM%3D&reserved=0
> >>
> >> In order to get a feeling about its usage (you need Docker or
> >> Podman), follow these simple copy&paste instructions:
> >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FSilvanoc%2Fmeta-iot2050%2Fblob%2Fmaster%2Fkas%2FBUILDING-SDK-CONTAINER.md%23running-the-sdk&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C70ab5dc719fe445169a008d8e3e01c27%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637509898980089588%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2FiIv247NEqKV%2F2osF6hL5S4QiA6T8yqd5TQpi3EbuGQ%3D&reserved=0
> >> Build instructions are available in the upper part of that
> >> document.
> >>
> >> Two new dependencies (umoci and skopeo -backporting from bullseye
> >> to buster works easily) are required to create containerized root
> >> filesystems (as specified in the documentation).
> >>
> >> Typical container image management actions (e.g. push an image to a
> >> container image regitry) are out of scope. Available tools (Docker,
> >> Skopeo, Buildah, Podman,...) should be used for these actions.
> >>
> >> A patch will follow this one to get the dependencies into the
> >> container images being provided by the project
> >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsiemens%2Fkas&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C70ab5dc719fe445169a008d8e3e01c27%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637509898980099579%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=eozG3LQR9pRGIcNiz9a0DckXhkBu8pcas9c%2F%2BWon9mU%3D&reserved=0
> >> (for `kas-container`, for example).
> >>
> >> Silvano Cirujano Cuesta (5):
> >> classes: add root filesystem containerizing class
> >> classes: add new image type 'container-img'
> >> sdk: add support for containerized sdk
> >> docs: document creation of container images
> >> ci: add container image testing configurations
> >>
> >> doc/user_manual.md | 127
> >> ++++++++++++++++++ meta-isar/conf/machine/container.conf |
> >> 5 + .../conf/multiconfig/container-bullseye.conf | 4 +
> >> .../conf/multiconfig/container-buster.conf | 4 +
> >> .../conf/multiconfig/container-focal.conf | 4 +
> >> .../conf/multiconfig/container-stretch.conf | 4 +
> >> meta/classes/container-img.bbclass | 18 +++
> >> .../classes/image-container-extension.bbclass | 81 +++++++++++
> >> meta/classes/image-sdk-extension.bbclass | 42 +++++-
> >> meta/classes/image.bbclass | 1 +
> >> scripts/ci_build.sh | 11 +-
> >> 11 files changed, 293 insertions(+), 8 deletions(-)
> >> create mode 100644 meta-isar/conf/machine/container.conf
> >> create mode 100644
> >> meta-isar/conf/multiconfig/container-bullseye.conf create mode
> >> 100644 meta-isar/conf/multiconfig/container-buster.conf create
> >> mode 100644 meta-isar/conf/multiconfig/container-focal.conf create
> >> mode 100644 meta-isar/conf/multiconfig/container-stretch.conf
> >> create mode 100644 meta/classes/container-img.bbclass create mode
> >> 100644 meta/classes/image-container-extension.bbclass
> > Thanks for adding CI cases, but Jenkins failed during
> > do_populate_sdk:
>
> Do you have the documented dependencies (umoci and skopeo backported
> from bullseye) in the CI system? I've tested the commands manually on
> my system using kas-container and they worked, but your setup is
> slightly different.
AFAIK the jenkins runner is debian buster running "shell" jobs. New
deps will not work here.
I did not review the changes, but hope those new deps have been added
to the docs.
Henning
> >
> > 14:17:17 + SDK_FORMAT=docker-archive
> > 14:17:17 + BB_ENV_WHITELIST=' SDK_FORMAT'
> > 14:17:17 + bitbake -c do_populate_sdk
> > mc:qemuarm-stretch:isar-image-base ...
> > 14:17:18 ERROR: Unable to parse
> > /workspace/build/isar_am_devel_fast/92/meta-isar/recipes-app/libhello/libhello.bb
> > 14:17:18 Traceback (most recent call last): 14:17:18 File
> > "/workspace/build/isar_am_devel_fast/92/bitbake/lib/bb/parse/ast.py",
> > line 382, in
> > multi_finalize(fn='/workspace/build/isar_am_devel_fast/92/meta-isar/recipes-app/libhello/libhello.bb',
> > d=<bb.data_smart.DataSmart object at 0x7f12917e1c50>): 14:17:18
> > try: 14:17:18 > finalize(fn, d) 14:17:18
> > except bb.parse.SkipRecipe as e: 14:17:18 File
> > "/workspace/build/isar_am_devel_fast/92/bitbake/lib/bb/parse/ast.py",
> > line 347, in
> > finalize(fn='/workspace/build/isar_am_devel_fast/92/meta-isar/recipes-app/libhello/libhello.bb',
> > d=<bb.data_smart.DataSmart object at 0x7f12917e1c50>,
> > variant=None): 14:17:18 14:17:18 >
> > bb.parse.siggen.finalise(fn, d, variant) 14:17:18 14:17:18 File
> > "/workspace/build/isar_am_devel_fast/92/bitbake/lib/bb/siggen.py",
> > line 182, in
> > SignatureGeneratorBasicHash.finalise(fn='/workspace/build/isar_am_devel_fast/92/meta-isar/recipes-app/libhello/libhello.bb',
> > d=<bb.data_smart.DataSmart object at 0x7f12917e1c50>,
> > variant=None): 14:17:18 try: 14:17:18 >
> > taskdeps = self._build_data(fn, d) 14:17:18 except
> > bb.parse.SkipRecipe: 14:17:18 File
> > "/workspace/build/isar_am_devel_fast/92/bitbake/lib/bb/siggen.py",
> > line 150, in
> > SignatureGeneratorBasicHash._build_data(fn='/workspace/build/isar_am_devel_fast/92/meta-isar/recipes-app/libhello/libhello.bb',
> > d=<bb.data_smart.DataSmart object at 0x7f12917e1c50>): 14:17:18
> > ignore_mismatch = ((d.getVar("BB_HASH_IGNORE_MISMATCH") or
> > '') == '1') 14:17:18 > tasklist, gendeps, lookupcache =
> > bb.data.generate_dependencies(d, self.basewhitelist) 14:17:18
> > 14:17:18 File
> > "/workspace/build/isar_am_devel_fast/92/bitbake/lib/bb/data.py",
> > line 371, in generate_dependencies(d=<bb.data_smart.DataSmart
> > object at 0x7f12917e1c50>, whitelist={'SHELL', 'DEPLOY_DIR',
> > 'PRSERV_DUMPFILE', 'SDKPKGSUFFIX', 'FILE_DIRNAME', 'TERM',
> > 'COREBASE', 'LICENSE_PATH', 'STAGING_DIR_HOST', 'SSTATE_PKGARCH',
> > 'PKGDATA_DIR', 'THISDIR', 'USER', 'EXTERNAL_TOOLCHAIN', 'PWD',
> > 'DL_DIR', 'LOGNAME', 'PATH', 'BB_LIMITEDDEPS', 'CCACHE',
> > 'BB_TASKHASH', 'CCACHE_DIR', 'PARALLEL_MAKE', 'BBSERVER',
> > 'PRSERV_LOCKDOWN', 'BBPATH', 'BB_WORKERCONTEXT', 'PRSERV_HOST',
> > 'WORKDIR', 'FILE', 'CCACHE_NOHASHDIR', 'PRSERV_DUMPDIR',
> > 'FILESEXTRAPATHS', 'HOME', 'BUILD_ARCH', 'STAMPCLEAN',
> > 'SSTATE_DIR', 'TMPDIR', 'FILESPATH', 'STAGING_DIR_TARGET'}):
> > 14:17:18 keys = set(key for key in d if not
> > key.startswith("__")) 14:17:18 > shelldeps = set(key for key
> > in d.getVar("__exportlist", False) if d.getVarFlag(key, "export",
> > False) and not d.getVarFlag(key, "unexport", False)) 14:17:18
> > varflagsexcl = d.getVar('BB_SIGNATURE_EXCLUDE_FLAGS') 14:17:18
> > TypeError: 'NoneType' object is not iterable
> >
> > Full log:
> > https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.isar-build.org%3A8080%2Fjob%2Fisar_am_devel_fast%2F92%2FconsoleFull&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C70ab5dc719fe445169a008d8e3e01c27%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637509898980099579%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=SS%2FYypaMBwrC9dvwJCeIrW9EjPqVfF2RXFghcfAn4Lc%3D&reserved=0
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v6 0/5] Add support for containerized root filesystems
2021-03-10 17:25 ` Henning Schild
@ 2021-03-10 17:30 ` Silvano Cirujano Cuesta
0 siblings, 0 replies; 11+ messages in thread
From: Silvano Cirujano Cuesta @ 2021-03-10 17:30 UTC (permalink / raw)
To: Henning Schild; +Cc: Anton Mikanovich, isar-users
On 10/03/2021 18:25, Henning Schild wrote:
> Am Wed, 10 Mar 2021 18:18:23 +0100
> schrieb "[ext] Silvano Cirujano Cuesta"
> <silvano.cirujano-cuesta@siemens.com>:
>
>> On 10/03/2021 17:18, Anton Mikanovich wrote:
>>> 09.03.2021 23:52, Silvano Cirujano Cuesta wrote:
>>>> This patch series provides support for containerized root
>>>> filesystems, for both target images and SDKs.
>>>>
>>>> For containerized target images the new image type `container-img`
>>>> has been added.
>>>>
>>>> For containerized SDKs the task `populate_sdk` has been extended.
>>>>
>>>> Containerized root filesystems are easy to distribute and run,
>>>> enabling this way following scenarios:
>>>> - Use ISAR to build container images meant to be run only in
>>>> containers.
>>>> - Use the same ISAR configuration to build images for
>>>> containers, VMs and bare-metal.
>>>> - Easy SDK distribution and "installation".
>>>> - Quickly testing certain applications in the workstation using
>>>> the target root filesystem.
>>>>
>>>> In order to build containerized target root filesystems
>>>> `IMAGE_TYPE` has to be `container-img`, additionally the container
>>>> image format can be selected with the variable
>>>> `CONTAINER_FORMATS`. The default format is `docker-archive`.
>>>>
>>>> In order to build containerized SDKs the variable `SDK_FORMAT` has
>>>> to provide any of the supported container formats (e.g.
>>>> `docker-archive`). The default format is the legacy
>>>> non-containerized: `tar_xz`.
>>>>
>>>> It also provides a sample machine, multiconfigs and ci-testing.
>>>>
>>>> More information about its usage is documented in the file
>>>> docs/user_manual.md.
>>>>
>>>> A PoC/demo of this functionality (only the SDK part) has been
>>>> created based on the project
>>>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsiemens%2Fmeta-iot2050&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C9aeeebc3bbcd43639e3408d8e3e97906%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637509939186866083%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2B2icpBtgUxWlC%2Byu%2B8mzTX%2B5f%2FUqwxx32eSm1OIt7jE%3D&reserved=0.
>>>> Jan Kiszka already tested and liked it! =>
>>>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsiemens%2Fmeta-iot2050%2Fissues%2F86%23issuecomment-768907845&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C9aeeebc3bbcd43639e3408d8e3e97906%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637509939186866083%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=KobF91tR3qTPXkVlm1siBENSZ8GI2yVgV%2FUR4wA%2Fa1g%3D&reserved=0
>>>>
>>>> Successful builds of both containerized target and SDK are
>>>> available on the same PoC project:
>>>> -
>>>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FSilvanoc%2Fmeta-iot2050%2Factions%2Fruns%2F558311580&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C9aeeebc3bbcd43639e3408d8e3e97906%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637509939186866083%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=SfmgjQlfFHt5GTWS1Bkh9fgYzdVus%2BxAQWL1HXU%2FEiM%3D&reserved=0
>>>> -
>>>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FSilvanoc%2Fmeta-iot2050%2Factions%2Fruns%2F558311581&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C9aeeebc3bbcd43639e3408d8e3e97906%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637509939186866083%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=tXRuiVa7uhPLnRG7Ll2oLrz%2BGP85bR6HEOOzTp7J%2Bxo%3D&reserved=0
>>>> and also the resulting images:
>>>> -
>>>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fusers%2FSilvanoc%2Fpackages%2Fcontainer%2Fpackage%2Fmeta-iot2050%252Fiot2050-debian-arm64&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C9aeeebc3bbcd43639e3408d8e3e97906%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637509939186876071%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=BKHSEDqOAHeum%2FGYdREIysEJ1JCTm68npNBZhgYkOr4%3D&reserved=0
>>>> -
>>>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fusers%2FSilvanoc%2Fpackages%2Fcontainer%2Fpackage%2Fmeta-iot2050%252Fiot2050-debian-sdk-arm64&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C9aeeebc3bbcd43639e3408d8e3e97906%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637509939186876071%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=EhFGqF5PnTNkeZub1%2FzDzFCClAT0cN4v2A95D85NEjo%3D&reserved=0
>>>>
>>>> In order to get a feeling about its usage (you need Docker or
>>>> Podman), follow these simple copy&paste instructions:
>>>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FSilvanoc%2Fmeta-iot2050%2Fblob%2Fmaster%2Fkas%2FBUILDING-SDK-CONTAINER.md%23running-the-sdk&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C9aeeebc3bbcd43639e3408d8e3e97906%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637509939186876071%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=GMOqu5mVVazU%2FnvpjiWLDoatp2Yi4XmYMkpZNLCMZts%3D&reserved=0
>>>> Build instructions are available in the upper part of that
>>>> document.
>>>>
>>>> Two new dependencies (umoci and skopeo -backporting from bullseye
>>>> to buster works easily) are required to create containerized root
>>>> filesystems (as specified in the documentation).
>>>>
>>>> Typical container image management actions (e.g. push an image to a
>>>> container image regitry) are out of scope. Available tools (Docker,
>>>> Skopeo, Buildah, Podman,...) should be used for these actions.
>>>>
>>>> A patch will follow this one to get the dependencies into the
>>>> container images being provided by the project
>>>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsiemens%2Fkas&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C9aeeebc3bbcd43639e3408d8e3e97906%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637509939186876071%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=0XCjNwZVf%2By0WL2C%2BRTRiVmgZT34dL0auBRFila0Uqs%3D&reserved=0
>>>> (for `kas-container`, for example).
>>>>
>>>> Silvano Cirujano Cuesta (5):
>>>> classes: add root filesystem containerizing class
>>>> classes: add new image type 'container-img'
>>>> sdk: add support for containerized sdk
>>>> docs: document creation of container images
>>>> ci: add container image testing configurations
>>>>
>>>> doc/user_manual.md | 127
>>>> ++++++++++++++++++ meta-isar/conf/machine/container.conf |
>>>> 5 + .../conf/multiconfig/container-bullseye.conf | 4 +
>>>> .../conf/multiconfig/container-buster.conf | 4 +
>>>> .../conf/multiconfig/container-focal.conf | 4 +
>>>> .../conf/multiconfig/container-stretch.conf | 4 +
>>>> meta/classes/container-img.bbclass | 18 +++
>>>> .../classes/image-container-extension.bbclass | 81 +++++++++++
>>>> meta/classes/image-sdk-extension.bbclass | 42 +++++-
>>>> meta/classes/image.bbclass | 1 +
>>>> scripts/ci_build.sh | 11 +-
>>>> 11 files changed, 293 insertions(+), 8 deletions(-)
>>>> create mode 100644 meta-isar/conf/machine/container.conf
>>>> create mode 100644
>>>> meta-isar/conf/multiconfig/container-bullseye.conf create mode
>>>> 100644 meta-isar/conf/multiconfig/container-buster.conf create
>>>> mode 100644 meta-isar/conf/multiconfig/container-focal.conf create
>>>> mode 100644 meta-isar/conf/multiconfig/container-stretch.conf
>>>> create mode 100644 meta/classes/container-img.bbclass create mode
>>>> 100644 meta/classes/image-container-extension.bbclass
>>> Thanks for adding CI cases, but Jenkins failed during
>>> do_populate_sdk:
>> Do you have the documented dependencies (umoci and skopeo backported
>> from bullseye) in the CI system? I've tested the commands manually on
>> my system using kas-container and they worked, but your setup is
>> slightly different.
> AFAIK the jenkins runner is debian buster running "shell" jobs. New
> deps will not work here.
> I did not review the changes, but hope those new deps have been added
> to the docs.
>
> Henning
Yes, it's documented both in the cover letter (look for 'umoci' in this e-mail) and in the user manual => https://github.com/Silvanoc/isar/commit/54c0af643f4011da1cf5171f1b8d8c29cf52769d#diff-266dae8a4f0ac77fb09535e9bbf9fa2516ea79ebc5c91b4781c6a537a6f8d64cR89
Silvano
>
>>> 14:17:17 + SDK_FORMAT=docker-archive
>>> 14:17:17 + BB_ENV_WHITELIST=' SDK_FORMAT'
>>> 14:17:17 + bitbake -c do_populate_sdk
>>> mc:qemuarm-stretch:isar-image-base ...
>>> 14:17:18 ERROR: Unable to parse
>>> /workspace/build/isar_am_devel_fast/92/meta-isar/recipes-app/libhello/libhello.bb
>>> 14:17:18 Traceback (most recent call last): 14:17:18 File
>>> "/workspace/build/isar_am_devel_fast/92/bitbake/lib/bb/parse/ast.py",
>>> line 382, in
>>> multi_finalize(fn='/workspace/build/isar_am_devel_fast/92/meta-isar/recipes-app/libhello/libhello.bb',
>>> d=<bb.data_smart.DataSmart object at 0x7f12917e1c50>): 14:17:18
>>> try: 14:17:18 > finalize(fn, d) 14:17:18
>>> except bb.parse.SkipRecipe as e: 14:17:18 File
>>> "/workspace/build/isar_am_devel_fast/92/bitbake/lib/bb/parse/ast.py",
>>> line 347, in
>>> finalize(fn='/workspace/build/isar_am_devel_fast/92/meta-isar/recipes-app/libhello/libhello.bb',
>>> d=<bb.data_smart.DataSmart object at 0x7f12917e1c50>,
>>> variant=None): 14:17:18 14:17:18 >
>>> bb.parse.siggen.finalise(fn, d, variant) 14:17:18 14:17:18 File
>>> "/workspace/build/isar_am_devel_fast/92/bitbake/lib/bb/siggen.py",
>>> line 182, in
>>> SignatureGeneratorBasicHash.finalise(fn='/workspace/build/isar_am_devel_fast/92/meta-isar/recipes-app/libhello/libhello.bb',
>>> d=<bb.data_smart.DataSmart object at 0x7f12917e1c50>,
>>> variant=None): 14:17:18 try: 14:17:18 >
>>> taskdeps = self._build_data(fn, d) 14:17:18 except
>>> bb.parse.SkipRecipe: 14:17:18 File
>>> "/workspace/build/isar_am_devel_fast/92/bitbake/lib/bb/siggen.py",
>>> line 150, in
>>> SignatureGeneratorBasicHash._build_data(fn='/workspace/build/isar_am_devel_fast/92/meta-isar/recipes-app/libhello/libhello.bb',
>>> d=<bb.data_smart.DataSmart object at 0x7f12917e1c50>): 14:17:18
>>> ignore_mismatch = ((d.getVar("BB_HASH_IGNORE_MISMATCH") or
>>> '') == '1') 14:17:18 > tasklist, gendeps, lookupcache =
>>> bb.data.generate_dependencies(d, self.basewhitelist) 14:17:18
>>> 14:17:18 File
>>> "/workspace/build/isar_am_devel_fast/92/bitbake/lib/bb/data.py",
>>> line 371, in generate_dependencies(d=<bb.data_smart.DataSmart
>>> object at 0x7f12917e1c50>, whitelist={'SHELL', 'DEPLOY_DIR',
>>> 'PRSERV_DUMPFILE', 'SDKPKGSUFFIX', 'FILE_DIRNAME', 'TERM',
>>> 'COREBASE', 'LICENSE_PATH', 'STAGING_DIR_HOST', 'SSTATE_PKGARCH',
>>> 'PKGDATA_DIR', 'THISDIR', 'USER', 'EXTERNAL_TOOLCHAIN', 'PWD',
>>> 'DL_DIR', 'LOGNAME', 'PATH', 'BB_LIMITEDDEPS', 'CCACHE',
>>> 'BB_TASKHASH', 'CCACHE_DIR', 'PARALLEL_MAKE', 'BBSERVER',
>>> 'PRSERV_LOCKDOWN', 'BBPATH', 'BB_WORKERCONTEXT', 'PRSERV_HOST',
>>> 'WORKDIR', 'FILE', 'CCACHE_NOHASHDIR', 'PRSERV_DUMPDIR',
>>> 'FILESEXTRAPATHS', 'HOME', 'BUILD_ARCH', 'STAMPCLEAN',
>>> 'SSTATE_DIR', 'TMPDIR', 'FILESPATH', 'STAGING_DIR_TARGET'}):
>>> 14:17:18 keys = set(key for key in d if not
>>> key.startswith("__")) 14:17:18 > shelldeps = set(key for key
>>> in d.getVar("__exportlist", False) if d.getVarFlag(key, "export",
>>> False) and not d.getVarFlag(key, "unexport", False)) 14:17:18
>>> varflagsexcl = d.getVar('BB_SIGNATURE_EXCLUDE_FLAGS') 14:17:18
>>> TypeError: 'NoneType' object is not iterable
>>>
>>> Full log:
>>> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.isar-build.org%3A8080%2Fjob%2Fisar_am_devel_fast%2F92%2FconsoleFull&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C9aeeebc3bbcd43639e3408d8e3e97906%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637509939186876071%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9PeSmqXdC07ubN7xFJ4uH90hxprBnG4NuR4z5EMa8Ug%3D&reserved=0
--
Siemens AG, T RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2021-03-10 17:30 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-09 20:52 [PATCH v6 0/5] Add support for containerized root filesystems Silvano Cirujano Cuesta
2021-03-09 20:52 ` [PATCH v6 1/5] classes: add root filesystem containerizing class Silvano Cirujano Cuesta
2021-03-09 20:52 ` [PATCH v6 2/5] classes: add new image type 'container-img' Silvano Cirujano Cuesta
2021-03-09 20:52 ` [PATCH v6 3/5] sdk: add support for containerized sdk Silvano Cirujano Cuesta
2021-03-09 20:52 ` [PATCH v6 4/5] docs: document creation of container images Silvano Cirujano Cuesta
2021-03-09 20:52 ` [PATCH v6 5/5] ci: add container image testing configurations Silvano Cirujano Cuesta
2021-03-10 16:18 ` [PATCH v6 0/5] Add support for containerized root filesystems Anton Mikanovich
2021-03-10 17:18 ` Silvano Cirujano Cuesta
2021-03-10 17:25 ` Henning Schild
2021-03-10 17:30 ` Silvano Cirujano Cuesta
2021-03-10 16:28 ` Henning Schild
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox