Re: [PATCH v8 1/5] classes: add root filesystem containerizing class

[Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>]

Apparently my mail client mangled my previous reply... :-/
Thanks to Jan for making me aware of it.

This one should be readable.

On 08/04/2021 19:55, Jan Kiszka wrote:
> On 29.03.21 17:56, [ext] Silvano Cirujano Cuesta wrote:
>> This class can be used to create container images which root filesystem
>> is that generated by the do_rootfs task.
>>
>> Containerized root filesystems have following possible use-cases:
>>  - Using ISAR as a container image builder.
>>  - Simplify distribution of runtime rootfs (binaries, libraries,
>>    configurations, ...) for application development or testing.
>>  - Distributing SDKs.
>>
>> Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>
>> ---
>>  .../classes/image-container-extension.bbclass | 82 +++++++++++++++++++
>>  meta/classes/image.bbclass                    |  1 +
>>  2 files changed, 83 insertions(+)
>>  create mode 100644 meta/classes/image-container-extension.bbclass
>>
>> diff --git a/meta/classes/image-container-extension.bbclass b/meta/classes/image-container-extension.bbclass
>> new file mode 100644
>> index 0000000..f693627
>> --- /dev/null
>> +++ b/meta/classes/image-container-extension.bbclass
>> @@ -0,0 +1,82 @@
>> +# This software is a part of ISAR.
>> +# Copyright (C) Siemens AG, 2021
>> +#
>> +# SPDX-License-Identifier: MIT
>> +#
>> +# This class extends the image.bbclass for containerizing the root filesystem.
>> +
>> +CONTAINER_FORMATS ?= "docker-archive"
>> +IMAGE_INSTALL += "isar-exclude-docs isar-disable-apt-cache"
>> +
> 
> Quirin tried latest Isar on meta-iot2050, and that now fails during
> setup of openjdk:
> 
> Setting up openjdk-11-jre-headless:arm64 (11.0.9.1+1-1~deb10u2) ...
> 
> 
> 
> update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/rmid
> to provide /usr/bin/rmid (rmid) in auto mode
> 
> 
> update-alternatives: error: error creating symbolic link
> '/usr/share/man/man1/rmid.1.gz.dpkg-tmp': No such file or directory
> 
> 
> 
> While watching the installation, I happened to see isar-exclude-docs
> suddenly being installed while not being selected explicitly,
> specifically not for the target image. I bet it's coming in via this
> class and its unconditional extension of the IMAGE_INSTALL list.

You're right, it was a late addition and I didn't realized that I was unconditionally adding it to image.bbclass.

This exclusion was only meant for container images, you don't want to have manpages and APT caches in a container image.

My fault, I'll provide a fix.

> 
> That leads to (at least) two questions:
>  - Why do we have isar-exclude-docs here, and also
>    isar-disable-apt-cache?
>  - Does isar-exclude-docs have some issue that prevents the usage
>    together with openjdk, or are those simply incompatible by design?

IMO if someone wants to get rid of manpages for very small devices, the recipe isar-exclude-docs should help on that.
The same applies to isar-disable-apt-cache.

My error just triggered a hidden issue: the incompatibility between the implementation of the isar-exclude-docs recipe and other components (for example, openjdk).
IMO this issue should get fix too (apart from the unconditional addition of the recipe).
Unless such a configuration is unsupported, but I think that's the scope of the other thread that you opened ;-)

> 
> In any case, we have a regression.

Yes, we do.

   Silvano

> 
> Jan
> 
>> +containerize_rootfs() {
>> +    local cmd="/bin/dash"
>> +    local empty_tag="empty"
>> +    local full_tag="latest"
>> +    local oci_img_dir="${WORKDIR}/oci-image"
>> +    local rootfs="$1"
>> +    local rootfs_id="$2"
>> +    local container_formats="$3"
>> +
>> +    # prepare OCI container image skeleton
>> +    bbdebug 1 "prepare OCI container image skeleton"
>> +    rm -rf "${oci_img_dir}"
>> +    sudo umoci init --layout "${oci_img_dir}"
>> +    sudo umoci new --image "${oci_img_dir}:${empty_tag}"
>> +    sudo umoci config --image "${oci_img_dir}:${empty_tag}" \
>> +        --config.cmd="${cmd}"
>> +    sudo umoci unpack --image "${oci_img_dir}:${empty_tag}" \
>> +        "${oci_img_dir}_unpacked"
>> +
>> +    # add root filesystem as the flesh of the skeleton
>> +    sudo cp -a "${rootfs}"/* "${oci_img_dir}_unpacked/rootfs/"
>> +    # clean-up temporary files
>> +    sudo find "${oci_img_dir}_unpacked/rootfs/tmp" -mindepth 1 -delete
>> +
>> +    # pack container image
>> +    bbdebug 1 "pack container image"
>> +    sudo umoci repack --image "${oci_img_dir}:${full_tag}" \
>> +        "${oci_img_dir}_unpacked"
>> +    sudo umoci remove --image "${oci_img_dir}:${empty_tag}"
>> +    sudo rm -rf "${oci_img_dir}_unpacked"
>> +
>> +    # no root needed anymore
>> +    sudo chown --recursive $(id -u):$(id -g) "${oci_img_dir}"
>> +
>> +    # convert the OCI container image to the desired format
>> +    image_name="isar-${rootfs_id}"
>> +    for image_type in ${CONTAINER_FORMATS} ; do
>> +        image_archive="${DEPLOY_DIR_IMAGE}/${rootfs_id}-${image_type}.tar"
>> +        bbdebug 1 "Creating container image type: ${image_type}"
>> +        case "${image_type}" in
>> +            "docker-archive" | "oci-archive")
>> +                if [ "${image_type}" = "oci-archive" ] ; then
>> +                    target="${image_type}:${image_archive}:latest"
>> +                else
>> +                    target="${image_type}:${image_archive}:${image_name}:latest"
>> +                fi
>> +                rm -f "${image_archive}" "${image_archive}.xz"
>> +                bbdebug 2 "Converting OCI image to ${image_type}"
>> +                skopeo --insecure-policy copy \
>> +                    "oci:${oci_img_dir}:${full_tag}" "${target}"
>> +                bbdebug 2 "Compressing image"
>> +                xz -T0 "${image_archive}"
>> +                ;;
>> +            "oci")
>> +                tar --create --xz --directory "${oci_img_dir}" \
>> +                    --file "${image_archive}.xz" .
>> +                ;;
>> +            "docker-daemon" | "containers-storage")
>> +                if [ -f /.dockerenv ] || [ -f /run/.containerenv ] ; then
>> +                    die "Adding the container image to a container runtime (${image_type}) not supported if running from a container (e.g. 'kas-container')"
>> +                fi
>> +                skopeo --insecure-policy copy \
>> +                    "oci:${oci_img_dir}:${full_tag}" \
>> +                    "${image_type}:${image_name}:latest"
>> +                ;;
>> +            *)
>> +                die "Unsupported format for containerize_rootfs: ${image_type}"
>> +                ;;
>> +        esac
>> +    done
>> +}
>> +
>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
>> index eddc444..ec93cab 100644
>> --- a/meta/classes/image.bbclass
>> +++ b/meta/classes/image.bbclass
>> @@ -76,6 +76,7 @@ inherit image-tools-extension
>>  inherit image-postproc-extension
>>  inherit image-locales-extension
>>  inherit image-account-extension
>> +inherit image-container-extension
>>  
>>  # Extra space for rootfs in MB
>>  ROOTFS_EXTRA ?= "64"
>>
> 
-- 
  Silvano Cirujano Cuesta
-- 
Siemens AG, T RDA IOT SES-DE
Corporate Competence Center Embedded Linux