Re: [RFC][PATCH 2/3] container-loader: Introduce helper to load container images into local registry

["'Jan Kiszka' via isar-users" <isar-users@googlegroups.com>]

On 12.07.24 14:11, Niedermayr, Benedikt (T CED OES-DE) wrote:
> On Tue, 2024-07-09 at 19:31 +0200, Jan Kiszka wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> This allows to write dpkg-raw recipes which packages archived container
>> images and load them into a local docker or podman registry on boot. The
>> scenario behind this is to pre-fill local registries in a way that still
>> permits live updates during runtime.
>>
>> The loader script only process images which are not yet available under
>> the same name and tag in the local registry. Also after loading, the
>> archived images stay on the local file system. This allows to perform
>> reloading in case the local registry should be emptied (e.g. reset to
>> factory state). To reduce the space those original images need, they are
>> compressed, by default with xz.
>>
>> Separate include files are available to cater the main container
>> engines, one for docker and one for podman.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>  .../container-loader/container-loader.inc     | 76 +++++++++++++++++++
>>  .../container-loader/docker-loader.inc        | 10 +++
>>  .../files/container-loader.service.tmpl       | 11 +++
>>  .../files/container-loader.sh.tmpl            | 13 ++++
>>  .../container-loader/podman-loader.inc        | 10 +++
>>  5 files changed, 120 insertions(+)
>>  create mode 100644 meta/recipes-support/container-loader/container-loader.inc
>>  create mode 100644 meta/recipes-support/container-loader/docker-loader.inc
>>  create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl
>>  create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl
>>  create mode 100644 meta/recipes-support/container-loader/podman-loader.inc
>>
>> diff --git a/meta/recipes-support/container-loader/container-loader.inc b/meta/recipes-
>> support/container-loader/container-loader.inc
>> new file mode 100644
>> index 00000000..8e352214
>> --- /dev/null
>> +++ b/meta/recipes-support/container-loader/container-loader.inc
>> @@ -0,0 +1,76 @@
>> +# This software is a part of ISAR.
>> +# Copyright (c) Siemens AG, 2024
>> +#
>> +# SPDX-License-Identifier: MIT
>> +
>> +FILESPATH:append := ":${FILE_DIRNAME}/files"
>> +
>> +inherit dpkg-raw
>> +
>> +SRC_URI += " \
>> +    file://container-loader.service.tmpl \
>> +    file://container-loader.sh.tmpl"
>> +
>> +TEMPLATE_FILES += " \
>> +    container-loader.service.tmpl \
>> +    container-loader.sh.tmpl"
>> +TEMPLATE_VARS += "CONTAINER_ENGINE"
>> +
>> +CONTAINER_COMPRESSION ?= "xz"
>> +
>> +DEBIAN_DEPENDS += " \
>> +    ${CONTAINER_ENGINE_PACKAGES} \
>> +    ${@', xz-utils' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
>> +       ', gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
>> +       ''}"
>> +
>> +CONTAINER_COMPRESSOR = "${@ \
>> +    'xz' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
>> +    'gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
>> +    ''}"
>> +
>> +python do_install() {
>> +    import os
>> +
>> +    workdir = d.getVar('WORKDIR')
>> +    D = d.getVar('D')
>> +    PN= d.getVar('PN')
>> +
>> +    image_list = open(D + "/usr/share/" + PN +"/image.list", "w")
>> +
>> +    src_uri = d.getVar('SRC_URI').split()
>> +    for uri in src_uri:
>> +        scheme, host, path, _, _, parm = bb.fetch.decodeurl(uri)
>> +        if scheme != "docker":
>> +            continue
>> +
>> +        image_name = host + (path if path != "/" else "")
>> +        unpacked_image = workdir + "/" + image_name.replace('/', '.')
>> +        dest_dir = D + "/usr/share/" + PN + "/images"
>> +        tar_image = dest_dir + "/" + image_name.replace('/', '.') + ".tar"
>> +        docker_ref = ":" + parm["tag"] if "tag" in parm else "latest"
>> +
>> +        cmd = f"skopeo copy dir:{unpacked_image} " \
>> +            f"docker-archive:{tar_image}:{image_name}{docker_ref}"
> 
> regarding performance,
> using "xz -T0" or pigz instead of gzip would increase performance and reduce build time a lot.
> 

(looks like you wanted to comment below)

> Benedikt
> 
>> +        bb.note(f"running: {cmd}")
>> +        bb.process.run(cmd)
>> +
>> +        cmd = f"{d.getVar('CONTAINER_COMPRESSOR')} {tar_image}"

We can tune CONTAINER_COMPRESSOR and its dependencies. I guess I will
also follow Felix suggestion to open-code decompression so that we avoid
shortcomings of podman's built-in decompression.

Jan

>> +        bb.note(f"running: {cmd}")
>> +        bb.process.run(cmd)
>> +
>> +        line = f"{os.path.basename(tar_image)}.{d.getVar('CONTAINER_COMPRESSION')} " + \
>> +            image_name + docker_ref
>> +        bb.note(f"adding '{line}' to image.list")
>> +        image_list.write(line + "\n")
>> +
>> +    image_list.close()
>> +
>> +    bb.utils.copyfile(workdir + "/container-loader.sh",
>> +                      D + "/usr/share/" + PN + "/container-loader.sh")
>> +}
>> +do_install[cleandirs] += "${D}/usr/share/${PN}/images"
>> +
>> +do_prepare_build:append() {
>> +    install -v -m 644 ${WORKDIR}/container-loader.service ${S}/debian/${PN}.service
>> +}
>> diff --git a/meta/recipes-support/container-loader/docker-loader.inc b/meta/recipes-
>> support/container-loader/docker-loader.inc
>> new file mode 100644
>> index 00000000..b864c854
>> --- /dev/null
>> +++ b/meta/recipes-support/container-loader/docker-loader.inc
>> @@ -0,0 +1,10 @@
>> +# This software is a part of ISAR.
>> +# Copyright (c) Siemens AG, 2024
>> +#
>> +# SPDX-License-Identifier: MIT
>> +
>> +require container-loader.inc
>> +
>> +CONTAINER_ENGINE = "docker"
>> +
>> +CONTAINER_ENGINE_PACKAGES ?= "docker.io, apparmor"
>> diff --git a/meta/recipes-support/container-loader/files/container-loader.service.tmpl
>> b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
>> new file mode 100644
>> index 00000000..afde55d3
>> --- /dev/null
>> +++ b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
>> @@ -0,0 +1,11 @@
>> +[Unit]
>> +Description=Load archived container images on boot
>> +After=${CONTAINER_ENGINE}.service
>> +
>> +[Service]
>> +Type=oneshot
>> +ExecStart=/usr/share/${PN}/container-loader.sh
>> +RemainAfterExit=true
>> +
>> +[Install]
>> +WantedBy=multi-user.target
>> diff --git a/meta/recipes-support/container-loader/files/container-loader.sh.tmpl b/meta/recipes-
>> support/container-loader/files/container-loader.sh.tmpl
>> new file mode 100755
>> index 00000000..31d27865
>> --- /dev/null
>> +++ b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl
>> @@ -0,0 +1,13 @@
>> +#!/bin/sh
>> +#
>> +# Copyright (c) Siemens AG, 2024
>> +#
>> +# SPDX-License-Identifier: MIT
>> +
>> +set -eu
>> +
>> +while read -r image ref; do
>> +    if [ -z "$(${CONTAINER_ENGINE} images -q "$ref")" ]; then
>> +        ${CONTAINER_ENGINE} load -i /usr/share/${PN}/images/"$image"
>> +    fi
>> +done < /usr/share/${PN}/image.list
>> diff --git a/meta/recipes-support/container-loader/podman-loader.inc b/meta/recipes-
>> support/container-loader/podman-loader.inc
>> new file mode 100644
>> index 00000000..d2c9a12d
>> --- /dev/null
>> +++ b/meta/recipes-support/container-loader/podman-loader.inc
>> @@ -0,0 +1,10 @@
>> +# This software is a part of ISAR.
>> +# Copyright (c) Siemens AG, 2024
>> +#
>> +# SPDX-License-Identifier: MIT
>> +
>> +require container-loader.inc
>> +
>> +CONTAINER_ENGINE = "podman"
>> +
>> +CONTAINER_ENGINE_PACKAGES ?= "podman"
> 

-- 
Siemens AG, Technology
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/919c79bd-c444-424c-9a85-627b444bd9b3%40siemens.com.