Re: [RFC][PATCH 2/3] container-loader: Introduce helper to load container images into local registry

["'Niedermayr, BENEDIKT' via isar-users" <isar-users@googlegroups.com>]

On Tue, 2024-07-09 at 19:31 +0200, Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> This allows to write dpkg-raw recipes which packages archived container
> images and load them into a local docker or podman registry on boot. The
> scenario behind this is to pre-fill local registries in a way that still
> permits live updates during runtime.
> 
> The loader script only process images which are not yet available under
> the same name and tag in the local registry. Also after loading, the
> archived images stay on the local file system. This allows to perform
> reloading in case the local registry should be emptied (e.g. reset to
> factory state). To reduce the space those original images need, they are
> compressed, by default with xz.
> 
> Separate include files are available to cater the main container
> engines, one for docker and one for podman.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  .../container-loader/container-loader.inc     | 76 +++++++++++++++++++
>  .../container-loader/docker-loader.inc        | 10 +++
>  .../files/container-loader.service.tmpl       | 11 +++
>  .../files/container-loader.sh.tmpl            | 13 ++++
>  .../container-loader/podman-loader.inc        | 10 +++
>  5 files changed, 120 insertions(+)
>  create mode 100644 meta/recipes-support/container-loader/container-loader.inc
>  create mode 100644 meta/recipes-support/container-loader/docker-loader.inc
>  create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl
>  create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl
>  create mode 100644 meta/recipes-support/container-loader/podman-loader.inc
> 
> diff --git a/meta/recipes-support/container-loader/container-loader.inc b/meta/recipes-
> support/container-loader/container-loader.inc
> new file mode 100644
> index 00000000..8e352214
> --- /dev/null
> +++ b/meta/recipes-support/container-loader/container-loader.inc
> @@ -0,0 +1,76 @@
> +# This software is a part of ISAR.
> +# Copyright (c) Siemens AG, 2024
> +#
> +# SPDX-License-Identifier: MIT
> +
> +FILESPATH:append := ":${FILE_DIRNAME}/files"
> +
> +inherit dpkg-raw
> +
> +SRC_URI += " \
> +    file://container-loader.service.tmpl \
> +    file://container-loader.sh.tmpl"
> +
> +TEMPLATE_FILES += " \
> +    container-loader.service.tmpl \
> +    container-loader.sh.tmpl"
> +TEMPLATE_VARS += "CONTAINER_ENGINE"
> +
> +CONTAINER_COMPRESSION ?= "xz"
> +
> +DEBIAN_DEPENDS += " \
> +    ${CONTAINER_ENGINE_PACKAGES} \
> +    ${@', xz-utils' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
> +       ', gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
> +       ''}"
> +
> +CONTAINER_COMPRESSOR = "${@ \
> +    'xz' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
> +    'gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
> +    ''}"
> +
> +python do_install() {
> +    import os
> +
> +    workdir = d.getVar('WORKDIR')
> +    D = d.getVar('D')
> +    PN= d.getVar('PN')
> +
> +    image_list = open(D + "/usr/share/" + PN +"/image.list", "w")
> +
> +    src_uri = d.getVar('SRC_URI').split()
> +    for uri in src_uri:
> +        scheme, host, path, _, _, parm = bb.fetch.decodeurl(uri)
> +        if scheme != "docker":
> +            continue
> +
> +        image_name = host + (path if path != "/" else "")
> +        unpacked_image = workdir + "/" + image_name.replace('/', '.')
> +        dest_dir = D + "/usr/share/" + PN + "/images"
> +        tar_image = dest_dir + "/" + image_name.replace('/', '.') + ".tar"
> +        docker_ref = ":" + parm["tag"] if "tag" in parm else "latest"
> +
> +        cmd = f"skopeo copy dir:{unpacked_image} " \
> +            f"docker-archive:{tar_image}:{image_name}{docker_ref}"

regarding performance,
using "xz -T0" or pigz instead of gzip would increase performance and reduce build time a lot.

Benedikt

> +        bb.note(f"running: {cmd}")
> +        bb.process.run(cmd)
> +
> +        cmd = f"{d.getVar('CONTAINER_COMPRESSOR')} {tar_image}"
> +        bb.note(f"running: {cmd}")
> +        bb.process.run(cmd)
> +
> +        line = f"{os.path.basename(tar_image)}.{d.getVar('CONTAINER_COMPRESSION')} " + \
> +            image_name + docker_ref
> +        bb.note(f"adding '{line}' to image.list")
> +        image_list.write(line + "\n")
> +
> +    image_list.close()
> +
> +    bb.utils.copyfile(workdir + "/container-loader.sh",
> +                      D + "/usr/share/" + PN + "/container-loader.sh")
> +}
> +do_install[cleandirs] += "${D}/usr/share/${PN}/images"
> +
> +do_prepare_build:append() {
> +    install -v -m 644 ${WORKDIR}/container-loader.service ${S}/debian/${PN}.service
> +}
> diff --git a/meta/recipes-support/container-loader/docker-loader.inc b/meta/recipes-
> support/container-loader/docker-loader.inc
> new file mode 100644
> index 00000000..b864c854
> --- /dev/null
> +++ b/meta/recipes-support/container-loader/docker-loader.inc
> @@ -0,0 +1,10 @@
> +# This software is a part of ISAR.
> +# Copyright (c) Siemens AG, 2024
> +#
> +# SPDX-License-Identifier: MIT
> +
> +require container-loader.inc
> +
> +CONTAINER_ENGINE = "docker"
> +
> +CONTAINER_ENGINE_PACKAGES ?= "docker.io, apparmor"
> diff --git a/meta/recipes-support/container-loader/files/container-loader.service.tmpl
> b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
> new file mode 100644
> index 00000000..afde55d3
> --- /dev/null
> +++ b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
> @@ -0,0 +1,11 @@
> +[Unit]
> +Description=Load archived container images on boot
> +After=${CONTAINER_ENGINE}.service
> +
> +[Service]
> +Type=oneshot
> +ExecStart=/usr/share/${PN}/container-loader.sh
> +RemainAfterExit=true
> +
> +[Install]
> +WantedBy=multi-user.target
> diff --git a/meta/recipes-support/container-loader/files/container-loader.sh.tmpl b/meta/recipes-
> support/container-loader/files/container-loader.sh.tmpl
> new file mode 100755
> index 00000000..31d27865
> --- /dev/null
> +++ b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl
> @@ -0,0 +1,13 @@
> +#!/bin/sh
> +#
> +# Copyright (c) Siemens AG, 2024
> +#
> +# SPDX-License-Identifier: MIT
> +
> +set -eu
> +
> +while read -r image ref; do
> +    if [ -z "$(${CONTAINER_ENGINE} images -q "$ref")" ]; then
> +        ${CONTAINER_ENGINE} load -i /usr/share/${PN}/images/"$image"
> +    fi
> +done < /usr/share/${PN}/image.list
> diff --git a/meta/recipes-support/container-loader/podman-loader.inc b/meta/recipes-
> support/container-loader/podman-loader.inc
> new file mode 100644
> index 00000000..d2c9a12d
> --- /dev/null
> +++ b/meta/recipes-support/container-loader/podman-loader.inc
> @@ -0,0 +1,10 @@
> +# This software is a part of ISAR.
> +# Copyright (c) Siemens AG, 2024
> +#
> +# SPDX-License-Identifier: MIT
> +
> +require container-loader.inc
> +
> +CONTAINER_ENGINE = "podman"
> +
> +CONTAINER_ENGINE_PACKAGES ?= "podman"

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/e7b2930dfd7ddac9130f998b5e639ab12adccd42.camel%40siemens.com.