[RFC][PATCH 0/3] Introduce container fetcher and pre-loader

[RFC][PATCH 0/3] Introduce container fetcher and pre-loader

['Jan Kiszka' via isar-users]

We noticed internally that there are at least 3 implementations for
solving the task of pre-loading container images into isar-built base
systems, may it be that the images are not publicly available or the
device is only poorly connected, if at all. This series tries to solve
the task generically - hopefully.

This still needs at least documentation and CI hook-up, possibly more,
but I'd like to invite colleagues and the community in general to early
feedback.

BTW, patch 1 also introduces a fairly nice pattern to register a real
bitbake fetcher, rather than meddling with SRC_URI like we do for apt://
right now. Therefore, I'm planning to refactor the apt fetcher soon as
well, maybe even with progress reporting...

Jan

Jan Kiszka (3):
  Introduce fetcher from container registries
  container-loader: Introduce helper to load container images into local
    registry
  meta-isar: Add demo packages for installing prebuilt containers

 kas/package/Kconfig                           | 19 ++++
 kas/package/pkg_prebuilt-docker-img.yaml      |  9 ++
 kas/package/pkg_prebuilt-podman-img.yaml      |  9 ++
 .../prebuilt-docker-img_0.1.bb                | 10 +++
 .../prebuilt-podman-img_0.1.bb                | 10 +++
 meta/classes/dpkg-base.bbclass                |  6 ++
 meta/lib/container_fetcher.py                 | 90 +++++++++++++++++++
 .../container-loader/container-loader.inc     | 76 ++++++++++++++++
 .../container-loader/docker-loader.inc        | 10 +++
 .../files/container-loader.service.tmpl       | 11 +++
 .../files/container-loader.sh.tmpl            | 13 +++
 .../container-loader/podman-loader.inc        | 10 +++
 12 files changed, 273 insertions(+)
 create mode 100644 kas/package/pkg_prebuilt-docker-img.yaml
 create mode 100644 kas/package/pkg_prebuilt-podman-img.yaml
 create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
 create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb
 create mode 100644 meta/lib/container_fetcher.py
 create mode 100644 meta/recipes-support/container-loader/container-loader.inc
 create mode 100644 meta/recipes-support/container-loader/docker-loader.inc
 create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl
 create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl
 create mode 100644 meta/recipes-support/container-loader/podman-loader.inc

-- 
2.43.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/cover.1720546299.git.jan.kiszka%40siemens.com.

[RFC][PATCH 1/3] Introduce fetcher from container registries

['Jan Kiszka' via isar-users]

From: Jan Kiszka <jan.kiszka@siemens.com>

This bitbake fetcher allows to pull container images from registries,
store them in the download cache and transfer them into the workdir of
recipes requesting the image. The format of the URL is

docker://[<host>/]<image>;digest=sha256:...[;tag=<tag>]

Fetching without digest is supported but will cause a warning, just like
downloading via wget without a checksum. If tag is left out, "latest" is
used.

The fetcher will try to pull all available variants of a multi-arch
image. If this is not needed, you can also directly specify the image
digest of a specific architecture.

Future versions may also introduce full unpacking of the fetched
container layers in workdir if use case come up.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg-base.bbclass |  6 +++
 meta/lib/container_fetcher.py  | 90 ++++++++++++++++++++++++++++++++++
 2 files changed, 96 insertions(+)
 create mode 100644 meta/lib/container_fetcher.py

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 789d6c74..d90b32a9 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -98,6 +98,12 @@ python() {
     if len(d.getVar('SRC_APT').strip()) > 0:
         bb.build.addtask('apt_unpack', 'do_patch', '', d)
         bb.build.addtask('cleanall_apt', 'do_cleanall', '', d)
+
+    # container docker fetcher
+    import container_fetcher
+    from bb.fetch2 import methods
+
+    methods.append(container_fetcher.Container())
 }
 
 do_apt_fetch() {
diff --git a/meta/lib/container_fetcher.py b/meta/lib/container_fetcher.py
new file mode 100644
index 00000000..8c713795
--- /dev/null
+++ b/meta/lib/container_fetcher.py
@@ -0,0 +1,90 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+import os
+import re
+from   bb.fetch2 import FetchMethod
+from   bb.fetch2 import logger
+from   bb.fetch2 import MissingChecksumEvent
+from   bb.fetch2 import NoChecksumError
+from   bb.fetch2 import runfetchcmd
+from   bb.progress import LineFilterProgressHandler
+
+class SkopeoProgressHandler(LineFilterProgressHandler):
+    def __init__(self, d):
+        super(SkopeoProgressHandler, self).__init__(d)
+        self._fire_progress(0)
+
+    def writeline(self, line):
+        match = re.findall(r'^Copying image .*\(([0-9]+/[0-9]+)\)$', line)
+        if match:
+            state = match[0].split('/')
+            progress = (int(state[0]) * 100) / int(state[1])
+            self.update(progress)
+        return True
+
+
+class Container(FetchMethod):
+    def supports(self, ud, d):
+        return ud.type in ['docker']
+
+    def urldata_init(self, ud, d):
+        ud.tag = "latest"
+        if "tag" in ud.parm:
+            ud.tag = ud.parm["tag"]
+
+        ud.digest = None
+        if "digest" in ud.parm:
+            ud.digest = ud.parm["digest"]
+
+        container_name = ud.host + (ud.path if ud.path != "/" else "")
+        ud.container_src = container_name + \
+            ("@" + ud.digest if ud.digest else ":" + ud.tag)
+        ud.localname = container_name.replace('/', '.')
+        ud.localfile = "container-images/" + ud.localname + \
+            "_" + (ud.digest.replace(":", "-") if ud.digest else ud.tag)
+
+    def download(self, ud, d):
+        progresshandler = SkopeoProgressHandler(d)
+        runfetchcmd(f"skopeo copy --preserve-digests --all docker://{ud.container_src} dir:{ud.localfile}",
+                    d, log=progresshandler)
+
+        if ud.digest:
+            return
+
+        checksum = bb.utils.sha256_file(ud.localpath + "/manifest.json")
+        checksum_line = f"SRC_URI = \"{ud.url};digest=sha256:{checksum}\""
+
+        strict = d.getVar("BB_STRICT_CHECKSUM") or "0"
+
+        # If strict checking enabled and neither sum defined, raise error
+        if strict == "1":
+            raise NoChecksumError(checksum_line)
+
+        checksum_event = {"sha256sum": checksum}
+        bb.event.fire(MissingChecksumEvent(ud.url, **checksum_event), d)
+
+        if strict == "ignore":
+            return
+
+        # Log missing digest so user can more easily add it
+        logger.warning(
+            f"Missing checksum for '{ud.localpath}', consider using this " \
+            f"SRC_URI in the recipe:\n{checksum_line}")
+
+    def unpack(self, ud, rootdir, d):
+        arch = d.getVar('DISTRO_ARCH')
+        variant_opt = ""
+        if arch == "armhf":
+            arch = "arm"
+            variant_opt = "--override-variant v7"
+        elif arch == "armel":
+            arch = "arm"
+            variant_opt = "--override-variant v6"
+        runfetchcmd(f"skopeo --override-arch {arch} {variant_opt} " \
+            f"copy dir:{ud.localpath} dir:{rootdir + '/' + ud.localname}", d)
+
+    def clean(self, ud, d):
+        bb.utils.remove(ud.localpath, recurse=True)
-- 
2.43.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/b2008f2718cc2040403819d0531e8bc2312505b4.1720546299.git.jan.kiszka%40siemens.com.

[RFC][PATCH 2/3] container-loader: Introduce helper to load container images into local registry

['Jan Kiszka' via isar-users]

From: Jan Kiszka <jan.kiszka@siemens.com>

This allows to write dpkg-raw recipes which packages archived container
images and load them into a local docker or podman registry on boot. The
scenario behind this is to pre-fill local registries in a way that still
permits live updates during runtime.

The loader script only process images which are not yet available under
the same name and tag in the local registry. Also after loading, the
archived images stay on the local file system. This allows to perform
reloading in case the local registry should be emptied (e.g. reset to
factory state). To reduce the space those original images need, they are
compressed, by default with xz.

Separate include files are available to cater the main container
engines, one for docker and one for podman.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 .../container-loader/container-loader.inc     | 76 +++++++++++++++++++
 .../container-loader/docker-loader.inc        | 10 +++
 .../files/container-loader.service.tmpl       | 11 +++
 .../files/container-loader.sh.tmpl            | 13 ++++
 .../container-loader/podman-loader.inc        | 10 +++
 5 files changed, 120 insertions(+)
 create mode 100644 meta/recipes-support/container-loader/container-loader.inc
 create mode 100644 meta/recipes-support/container-loader/docker-loader.inc
 create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl
 create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl
 create mode 100644 meta/recipes-support/container-loader/podman-loader.inc

diff --git a/meta/recipes-support/container-loader/container-loader.inc b/meta/recipes-support/container-loader/container-loader.inc
new file mode 100644
index 00000000..8e352214
--- /dev/null
+++ b/meta/recipes-support/container-loader/container-loader.inc
@@ -0,0 +1,76 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+FILESPATH:append := ":${FILE_DIRNAME}/files"
+
+inherit dpkg-raw
+
+SRC_URI += " \
+    file://container-loader.service.tmpl \
+    file://container-loader.sh.tmpl"
+
+TEMPLATE_FILES += " \
+    container-loader.service.tmpl \
+    container-loader.sh.tmpl"
+TEMPLATE_VARS += "CONTAINER_ENGINE"
+
+CONTAINER_COMPRESSION ?= "xz"
+
+DEBIAN_DEPENDS += " \
+    ${CONTAINER_ENGINE_PACKAGES} \
+    ${@', xz-utils' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
+       ', gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
+       ''}"
+
+CONTAINER_COMPRESSOR = "${@ \
+    'xz' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
+    'gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
+    ''}"
+
+python do_install() {
+    import os
+
+    workdir = d.getVar('WORKDIR')
+    D = d.getVar('D')
+    PN= d.getVar('PN')
+
+    image_list = open(D + "/usr/share/" + PN +"/image.list", "w")
+
+    src_uri = d.getVar('SRC_URI').split()
+    for uri in src_uri:
+        scheme, host, path, _, _, parm = bb.fetch.decodeurl(uri)
+        if scheme != "docker":
+            continue
+
+        image_name = host + (path if path != "/" else "")
+        unpacked_image = workdir + "/" + image_name.replace('/', '.')
+        dest_dir = D + "/usr/share/" + PN + "/images"
+        tar_image = dest_dir + "/" + image_name.replace('/', '.') + ".tar"
+        docker_ref = ":" + parm["tag"] if "tag" in parm else "latest"
+
+        cmd = f"skopeo copy dir:{unpacked_image} " \
+            f"docker-archive:{tar_image}:{image_name}{docker_ref}"
+        bb.note(f"running: {cmd}")
+        bb.process.run(cmd)
+
+        cmd = f"{d.getVar('CONTAINER_COMPRESSOR')} {tar_image}"
+        bb.note(f"running: {cmd}")
+        bb.process.run(cmd)
+
+        line = f"{os.path.basename(tar_image)}.{d.getVar('CONTAINER_COMPRESSION')} " + \
+            image_name + docker_ref
+        bb.note(f"adding '{line}' to image.list")
+        image_list.write(line + "\n")
+
+    image_list.close()
+
+    bb.utils.copyfile(workdir + "/container-loader.sh",
+                      D + "/usr/share/" + PN + "/container-loader.sh")
+}
+do_install[cleandirs] += "${D}/usr/share/${PN}/images"
+
+do_prepare_build:append() {
+    install -v -m 644 ${WORKDIR}/container-loader.service ${S}/debian/${PN}.service
+}
diff --git a/meta/recipes-support/container-loader/docker-loader.inc b/meta/recipes-support/container-loader/docker-loader.inc
new file mode 100644
index 00000000..b864c854
--- /dev/null
+++ b/meta/recipes-support/container-loader/docker-loader.inc
@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+require container-loader.inc
+
+CONTAINER_ENGINE = "docker"
+
+CONTAINER_ENGINE_PACKAGES ?= "docker.io, apparmor"
diff --git a/meta/recipes-support/container-loader/files/container-loader.service.tmpl b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
new file mode 100644
index 00000000..afde55d3
--- /dev/null
+++ b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
@@ -0,0 +1,11 @@
+[Unit]
+Description=Load archived container images on boot
+After=${CONTAINER_ENGINE}.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/share/${PN}/container-loader.sh
+RemainAfterExit=true
+
+[Install]
+WantedBy=multi-user.target
diff --git a/meta/recipes-support/container-loader/files/container-loader.sh.tmpl b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl
new file mode 100755
index 00000000..31d27865
--- /dev/null
+++ b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl
@@ -0,0 +1,13 @@
+#!/bin/sh
+#
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+set -eu
+
+while read -r image ref; do
+    if [ -z "$(${CONTAINER_ENGINE} images -q "$ref")" ]; then
+        ${CONTAINER_ENGINE} load -i /usr/share/${PN}/images/"$image"
+    fi
+done < /usr/share/${PN}/image.list
diff --git a/meta/recipes-support/container-loader/podman-loader.inc b/meta/recipes-support/container-loader/podman-loader.inc
new file mode 100644
index 00000000..d2c9a12d
--- /dev/null
+++ b/meta/recipes-support/container-loader/podman-loader.inc
@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+require container-loader.inc
+
+CONTAINER_ENGINE = "podman"
+
+CONTAINER_ENGINE_PACKAGES ?= "podman"
-- 
2.43.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/ba72fae1b0dff34ff4474a2cd53939a6c4fd3279.1720546299.git.jan.kiszka%40siemens.com.

[RFC][PATCH 3/3] meta-isar: Add demo packages for installing prebuilt containers

['Jan Kiszka' via isar-users]

From: Jan Kiszka <jan.kiszka@siemens.com>

One recipe for docker, one for podman. Both pull from a registry that,
in contrast to infamous dockerhub, should not throttle CI jobs running
these frequently for testing purposes.

The podman variant of the recipe is intentionally leaving out the digest
to trigger the related warning of the container fetcher.

These demos also come with kas integration.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 kas/package/Kconfig                           | 19 +++++++++++++++++++
 kas/package/pkg_prebuilt-docker-img.yaml      |  9 +++++++++
 kas/package/pkg_prebuilt-podman-img.yaml      |  9 +++++++++
 .../prebuilt-docker-img_0.1.bb                | 10 ++++++++++
 .../prebuilt-podman-img_0.1.bb                | 10 ++++++++++
 5 files changed, 57 insertions(+)
 create mode 100644 kas/package/pkg_prebuilt-docker-img.yaml
 create mode 100644 kas/package/pkg_prebuilt-podman-img.yaml
 create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
 create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb

diff --git a/kas/package/Kconfig b/kas/package/Kconfig
index 35ba7cf1..395c3a08 100644
--- a/kas/package/Kconfig
+++ b/kas/package/Kconfig
@@ -146,6 +146,25 @@ config KAS_INCLUDE_PACKAGE_ISAR_CI_SSH_SETUP
         default "kas/package/pkg_sshd-regen-keys.yaml"
         depends on PACKAGE_ISAR_CI_SSH_SETUP
 
+
+config PACKAGE_PREBUILT_DOCKER_IMG
+        bool "prebuilt-docker-img"
+        default y
+
+config KAS_INCLUDE_PACKAGE_PREBUILT_DOCKER_IMG
+        string
+        default "kas/package/pkg_prebuilt-docker-img.yaml"
+        depends on PACKAGE_PREBUILT_DOCKER_IMG
+
+config PACKAGE_PREBUILT_PODMAN_IMG
+        bool "prebuilt-podman-img"
+        default y
+
+config KAS_INCLUDE_PACKAGE_PREBUILT_PODMAN_IMG
+        string
+        default "kas/package/pkg_prebuilt-podman-img.yaml"
+        depends on PACKAGE_PREBUILT_PODMAN_IMG
+
 endmenu
 
 config KAS_IMAGE_PREINSTALL
diff --git a/kas/package/pkg_prebuilt-docker-img.yaml b/kas/package/pkg_prebuilt-docker-img.yaml
new file mode 100644
index 00000000..df96a484
--- /dev/null
+++ b/kas/package/pkg_prebuilt-docker-img.yaml
@@ -0,0 +1,9 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+
+header:
+  version: 14
+
+local_conf_header:
+  package-prebuilt-docker-img: |
+    IMAGE_INSTALL:append = " prebuilt-docker-img"
diff --git a/kas/package/pkg_prebuilt-podman-img.yaml b/kas/package/pkg_prebuilt-podman-img.yaml
new file mode 100644
index 00000000..d0b8da1c
--- /dev/null
+++ b/kas/package/pkg_prebuilt-podman-img.yaml
@@ -0,0 +1,9 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+
+header:
+  version: 14
+
+local_conf_header:
+  package-prebuilt-podman-img: |
+    IMAGE_INSTALL:append = " prebuilt-podman-img"
diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
new file mode 100644
index 00000000..3f337d92
--- /dev/null
+++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+require recipes-support/container-loader/docker-loader.inc
+
+SRC_URI += "\
+    docker://quay.io/libpod/alpine;digest=sha256:fa93b01658e3a5a1686dc3ae55f170d8de487006fb53a28efcd12ab0710a2e5f;tag=3.10.2 \
+    "
diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb
new file mode 100644
index 00000000..e671a494
--- /dev/null
+++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb
@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+require recipes-support/container-loader/podman-loader.inc
+
+SRC_URI += "\
+    docker://quay.io/libpod/alpine;tag=latest \
+    "
-- 
2.43.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/e49de13980681ebcef06ae21a806a939415ca0b7.1720546299.git.jan.kiszka%40siemens.com.

Re: [RFC][PATCH 1/3] Introduce fetcher from container registries

['Florian Bezdeka' via isar-users]

On Tue, 2024-07-09 at 19:31 +0200, 'Jan Kiszka' via isar-users wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> This bitbake fetcher allows to pull container images from registries,
> store them in the download cache and transfer them into the workdir of
> recipes requesting the image. The format of the URL is
> 
> docker://[<host>/]<image>;digest=sha256:...[;tag=<tag>]

The schema - docker:// - is defined by / required by skopeo below,
right? 

Would it be possible to use something like container:// to avoid the
product name in all the user recipes?

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/41792a6ebde246fd404a4df825ab04a8a7a26588.camel%40siemens.com.

Re: [RFC][PATCH 1/3] Introduce fetcher from container registries

['Jan Kiszka' via isar-users]

On 10.07.24 09:55, Florian Bezdeka wrote:
> On Tue, 2024-07-09 at 19:31 +0200, 'Jan Kiszka' via isar-users wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> This bitbake fetcher allows to pull container images from registries,
>> store them in the download cache and transfer them into the workdir of
>> recipes requesting the image. The format of the URL is
>>
>> docker://[<host>/]<image>;digest=sha256:...[;tag=<tag>]
> 
> The schema - docker:// - is defined by / required by skopeo below,
> right? 
> 
> Would it be possible to use something like container:// to avoid the
> product name in all the user recipes?
> 

I'm re-using skopeo's schema here, right:

https://manpages.debian.org/bookworm/skopeo/skopeo.1.en.html

It would be technically possible to use container:// instead or as well
and translate that internally. But would that really be helpful / intuitive?

Jan

-- 
Siemens AG, Technology
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/1f6eb7b8-7443-4f9b-bafe-4cfed5c6de41%40siemens.com.

Re: [RFC][PATCH 2/3] container-loader: Introduce helper to load container images into local registry

['MOESSBAUER, Felix' via isar-users]

On Tue, 2024-07-09 at 19:31 +0200, Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> This allows to write dpkg-raw recipes which packages archived
> container
> images and load them into a local docker or podman registry on boot.
> The
> scenario behind this is to pre-fill local registries in a way that
> still
> permits live updates during runtime.
> 
> The loader script only process images which are not yet available
> under
> the same name and tag in the local registry. Also after loading, the
> archived images stay on the local file system. This allows to perform
> reloading in case the local registry should be emptied (e.g. reset to
> factory state). To reduce the space those original images need, they
> are
> compressed, by default with xz.

Hi, can we switch to zstd please? That's much quicker and needs a
fraction of the memory for decompression. On small devices,
decompressing large xz files is sometimes simply not possible due to
OOM.

> 
> Separate include files are available to cater the main container
> engines, one for docker and one for podman.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  .../container-loader/container-loader.inc     | 76
> +++++++++++++++++++
>  .../container-loader/docker-loader.inc        | 10 +++
>  .../files/container-loader.service.tmpl       | 11 +++
>  .../files/container-loader.sh.tmpl            | 13 ++++
>  .../container-loader/podman-loader.inc        | 10 +++
>  5 files changed, 120 insertions(+)
>  create mode 100644 meta/recipes-support/container-loader/container-
> loader.inc
>  create mode 100644 meta/recipes-support/container-loader/docker-
> loader.inc
>  create mode 100644 meta/recipes-support/container-
> loader/files/container-loader.service.tmpl
>  create mode 100755 meta/recipes-support/container-
> loader/files/container-loader.sh.tmpl
>  create mode 100644 meta/recipes-support/container-loader/podman-
> loader.inc
> 
> diff --git a/meta/recipes-support/container-loader/container-
> loader.inc b/meta/recipes-support/container-loader/container-
> loader.inc
> new file mode 100644
> index 00000000..8e352214
> --- /dev/null
> +++ b/meta/recipes-support/container-loader/container-loader.inc
> @@ -0,0 +1,76 @@
> +# This software is a part of ISAR.
> +# Copyright (c) Siemens AG, 2024
> +#
> +# SPDX-License-Identifier: MIT
> +
> +FILESPATH:append := ":${FILE_DIRNAME}/files"
> +
> +inherit dpkg-raw
> +
> +SRC_URI += " \
> +    file://container-loader.service.tmpl \
> +    file://container-loader.sh.tmpl"
> +
> +TEMPLATE_FILES += " \
> +    container-loader.service.tmpl \
> +    container-loader.sh.tmpl"
> +TEMPLATE_VARS += "CONTAINER_ENGINE"
> +
> +CONTAINER_COMPRESSION ?= "xz"
> +
> +DEBIAN_DEPENDS += " \
> +    ${CONTAINER_ENGINE_PACKAGES} \
> +    ${@', xz-utils' if d.getVar('CONTAINER_COMPRESSION') == 'xz'
> else \
> +       ', gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
> +       ''}"
> +
> +CONTAINER_COMPRESSOR = "${@ \
> +    'xz' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
> +    'gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
> +    ''}"
> +
> +python do_install() {
> +    import os
> +
> +    workdir = d.getVar('WORKDIR')
> +    D = d.getVar('D')
> +    PN= d.getVar('PN')
> +
> +    image_list = open(D + "/usr/share/" + PN +"/image.list", "w")
> +
> +    src_uri = d.getVar('SRC_URI').split()
> +    for uri in src_uri:
> +        scheme, host, path, _, _, parm = bb.fetch.decodeurl(uri)
> +        if scheme != "docker":
> +            continue
> +
> +        image_name = host + (path if path != "/" else "")
> +        unpacked_image = workdir + "/" + image_name.replace('/',
> '.')
> +        dest_dir = D + "/usr/share/" + PN + "/images"
> +        tar_image = dest_dir + "/" + image_name.replace('/', '.') +
> ".tar"
> +        docker_ref = ":" + parm["tag"] if "tag" in parm else
> "latest"
> +
> +        cmd = f"skopeo copy dir:{unpacked_image} " \
> +            f"docker-archive:{tar_image}:{image_name}{docker_ref}"

Why not oci-archive? In oci-archive, the layers are compressed. Or do
we explicitly not want to have the layers themselves compressed as we
compress the whole artifact anyways?

> +        bb.note(f"running: {cmd}")
> +        bb.process.run(cmd)
> +
> +        cmd = f"{d.getVar('CONTAINER_COMPRESSOR')} {tar_image}"
> +        bb.note(f"running: {cmd}")
> +        bb.process.run(cmd)
> +
> +        line =
> f"{os.path.basename(tar_image)}.{d.getVar('CONTAINER_COMPRESSION')} "
> + \
> +            image_name + docker_ref
> +        bb.note(f"adding '{line}' to image.list")
> +        image_list.write(line + "\n")
> +
> +    image_list.close()
> +
> +    bb.utils.copyfile(workdir + "/container-loader.sh",
> +                      D + "/usr/share/" + PN +

Shouldn't that be BPN instead of PN?

> "/container-loader.sh")
> +}
> +do_install[cleandirs] += "${D}/usr/share/${PN}/images"

Same here.

> +
> +do_prepare_build:append() {
> +    install -v -m 644 ${WORKDIR}/container-loader.service
> ${S}/debian/${PN}.service
> +}
> diff --git a/meta/recipes-support/container-loader/docker-loader.inc
> b/meta/recipes-support/container-loader/docker-loader.inc
> new file mode 100644
> index 00000000..b864c854
> --- /dev/null
> +++ b/meta/recipes-support/container-loader/docker-loader.inc
> @@ -0,0 +1,10 @@
> +# This software is a part of ISAR.
> +# Copyright (c) Siemens AG, 2024
> +#
> +# SPDX-License-Identifier: MIT
> +
> +require container-loader.inc
> +
> +CONTAINER_ENGINE = "docker"
> +
> +CONTAINER_ENGINE_PACKAGES ?= "docker.io, apparmor"
> diff --git a/meta/recipes-support/container-loader/files/container-
> loader.service.tmpl b/meta/recipes-support/container-
> loader/files/container-loader.service.tmpl
> new file mode 100644
> index 00000000..afde55d3
> --- /dev/null
> +++ b/meta/recipes-support/container-loader/files/container-
> loader.service.tmpl
> @@ -0,0 +1,11 @@
> +[Unit]
> +Description=Load archived container images on boot
> +After=${CONTAINER_ENGINE}.service

We need an Requires=${CONTAINER_ENGINE}.service as well. Otherwise just
the temporal order is defined, but not the logical order.

> +
> +[Service]
> +Type=oneshot
> +ExecStart=/usr/share/${PN}/container-loader.sh
> +RemainAfterExit=true

In case of rootless podman, the user that executes the script matters.
But I don't know if we want to consider this use-case by now.

> +
> +[Install]
> +WantedBy=multi-user.target
> diff --git a/meta/recipes-support/container-loader/files/container-
> loader.sh.tmpl b/meta/recipes-support/container-
> loader/files/container-loader.sh.tmpl
> new file mode 100755
> index 00000000..31d27865
> --- /dev/null
> +++ b/meta/recipes-support/container-loader/files/container-
> loader.sh.tmpl
> @@ -0,0 +1,13 @@
> +#!/bin/sh
> +#
> +# Copyright (c) Siemens AG, 2024
> +#
> +# SPDX-License-Identifier: MIT
> +
> +set -eu
> +
> +while read -r image ref; do
> +    if [ -z "$(${CONTAINER_ENGINE} images -q "$ref")" ]; then
> +        ${CONTAINER_ENGINE} load -i /usr/share/${PN}/images/"$image"

That does not work with all compression formats. Better decompress the
image manually and pass to stdin of the <tool> load. e.g.

cat $image | unzstd | podman load -

Also note, that podman needs tons of temporary space to load a
compressed image (or compressed layers), so you might want to set e.g.
TMPDIR=/var/tmp as /tmp could be a small memory backed fs.

Felix

> +    fi
> +done < /usr/share/${PN}/image.list
> diff --git a/meta/recipes-support/container-loader/podman-loader.inc
> b/meta/recipes-support/container-loader/podman-loader.inc
> new file mode 100644
> index 00000000..d2c9a12d
> --- /dev/null
> +++ b/meta/recipes-support/container-loader/podman-loader.inc
> @@ -0,0 +1,10 @@
> +# This software is a part of ISAR.
> +# Copyright (c) Siemens AG, 2024
> +#
> +# SPDX-License-Identifier: MIT
> +
> +require container-loader.inc
> +
> +CONTAINER_ENGINE = "podman"
> +
> +CONTAINER_ENGINE_PACKAGES ?= "podman"

-- 
Siemens AG, Technology
Linux Expert Center


-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/2ce98b6143b9cc6bee19332edd9adfcf8da96fa7.camel%40siemens.com.

Re: [RFC][PATCH 2/3] container-loader: Introduce helper to load container images into local registry

['Jan Kiszka' via isar-users]

On 10.07.24 14:46, Moessbauer, Felix (T CED OES-DE) wrote:
> On Tue, 2024-07-09 at 19:31 +0200, Jan Kiszka wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> This allows to write dpkg-raw recipes which packages archived
>> container
>> images and load them into a local docker or podman registry on boot.
>> The
>> scenario behind this is to pre-fill local registries in a way that
>> still
>> permits live updates during runtime.
>>
>> The loader script only process images which are not yet available
>> under
>> the same name and tag in the local registry. Also after loading, the
>> archived images stay on the local file system. This allows to perform
>> reloading in case the local registry should be emptied (e.g. reset to
>> factory state). To reduce the space those original images need, they
>> are
>> compressed, by default with xz.
> 
> Hi, can we switch to zstd please? That's much quicker and needs a
> fraction of the memory for decompression. On small devices,
> decompressing large xz files is sometimes simply not possible due to
> OOM.

Can be added - once Debian is on a docker version that supports it :)

> 
>>
>> Separate include files are available to cater the main container
>> engines, one for docker and one for podman.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>  .../container-loader/container-loader.inc     | 76
>> +++++++++++++++++++
>>  .../container-loader/docker-loader.inc        | 10 +++
>>  .../files/container-loader.service.tmpl       | 11 +++
>>  .../files/container-loader.sh.tmpl            | 13 ++++
>>  .../container-loader/podman-loader.inc        | 10 +++
>>  5 files changed, 120 insertions(+)
>>  create mode 100644 meta/recipes-support/container-loader/container-
>> loader.inc
>>  create mode 100644 meta/recipes-support/container-loader/docker-
>> loader.inc
>>  create mode 100644 meta/recipes-support/container-
>> loader/files/container-loader.service.tmpl
>>  create mode 100755 meta/recipes-support/container-
>> loader/files/container-loader.sh.tmpl
>>  create mode 100644 meta/recipes-support/container-loader/podman-
>> loader.inc
>>
>> diff --git a/meta/recipes-support/container-loader/container-
>> loader.inc b/meta/recipes-support/container-loader/container-
>> loader.inc
>> new file mode 100644
>> index 00000000..8e352214
>> --- /dev/null
>> +++ b/meta/recipes-support/container-loader/container-loader.inc
>> @@ -0,0 +1,76 @@
>> +# This software is a part of ISAR.
>> +# Copyright (c) Siemens AG, 2024
>> +#
>> +# SPDX-License-Identifier: MIT
>> +
>> +FILESPATH:append := ":${FILE_DIRNAME}/files"
>> +
>> +inherit dpkg-raw
>> +
>> +SRC_URI += " \
>> +    file://container-loader.service.tmpl \
>> +    file://container-loader.sh.tmpl"
>> +
>> +TEMPLATE_FILES += " \
>> +    container-loader.service.tmpl \
>> +    container-loader.sh.tmpl"
>> +TEMPLATE_VARS += "CONTAINER_ENGINE"
>> +
>> +CONTAINER_COMPRESSION ?= "xz"
>> +
>> +DEBIAN_DEPENDS += " \
>> +    ${CONTAINER_ENGINE_PACKAGES} \
>> +    ${@', xz-utils' if d.getVar('CONTAINER_COMPRESSION') == 'xz'
>> else \
>> +       ', gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
>> +       ''}"
>> +
>> +CONTAINER_COMPRESSOR = "${@ \
>> +    'xz' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
>> +    'gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
>> +    ''}"
>> +
>> +python do_install() {
>> +    import os
>> +
>> +    workdir = d.getVar('WORKDIR')
>> +    D = d.getVar('D')
>> +    PN= d.getVar('PN')
>> +
>> +    image_list = open(D + "/usr/share/" + PN +"/image.list", "w")
>> +
>> +    src_uri = d.getVar('SRC_URI').split()
>> +    for uri in src_uri:
>> +        scheme, host, path, _, _, parm = bb.fetch.decodeurl(uri)
>> +        if scheme != "docker":
>> +            continue
>> +
>> +        image_name = host + (path if path != "/" else "")
>> +        unpacked_image = workdir + "/" + image_name.replace('/',
>> '.')
>> +        dest_dir = D + "/usr/share/" + PN + "/images"
>> +        tar_image = dest_dir + "/" + image_name.replace('/', '.') +
>> ".tar"
>> +        docker_ref = ":" + parm["tag"] if "tag" in parm else
>> "latest"
>> +
>> +        cmd = f"skopeo copy dir:{unpacked_image} " \
>> +            f"docker-archive:{tar_image}:{image_name}{docker_ref}"
> 
> Why not oci-archive? In oci-archive, the layers are compressed. Or do
> we explicitly not want to have the layers themselves compressed as we
> compress the whole artifact anyways?

Not tested if we aren't causing any conversion issues towards docker &
friends - with versions of docker we currently use. Are you sure, for
bookworm and bullseye?

> 
>> +        bb.note(f"running: {cmd}")
>> +        bb.process.run(cmd)
>> +
>> +        cmd = f"{d.getVar('CONTAINER_COMPRESSOR')} {tar_image}"
>> +        bb.note(f"running: {cmd}")
>> +        bb.process.run(cmd)
>> +
>> +        line =
>> f"{os.path.basename(tar_image)}.{d.getVar('CONTAINER_COMPRESSION')} "
>> + \
>> +            image_name + docker_ref
>> +        bb.note(f"adding '{line}' to image.list")
>> +        image_list.write(line + "\n")
>> +
>> +    image_list.close()
>> +
>> +    bb.utils.copyfile(workdir + "/container-loader.sh",
>> +                      D + "/usr/share/" + PN +
> 
> Shouldn't that be BPN instead of PN?

Oh, yes, thanks.

> 
>> "/container-loader.sh")
>> +}
>> +do_install[cleandirs] += "${D}/usr/share/${PN}/images"
> 
> Same here.
> 
>> +
>> +do_prepare_build:append() {
>> +    install -v -m 644 ${WORKDIR}/container-loader.service
>> ${S}/debian/${PN}.service
>> +}
>> diff --git a/meta/recipes-support/container-loader/docker-loader.inc
>> b/meta/recipes-support/container-loader/docker-loader.inc
>> new file mode 100644
>> index 00000000..b864c854
>> --- /dev/null
>> +++ b/meta/recipes-support/container-loader/docker-loader.inc
>> @@ -0,0 +1,10 @@
>> +# This software is a part of ISAR.
>> +# Copyright (c) Siemens AG, 2024
>> +#
>> +# SPDX-License-Identifier: MIT
>> +
>> +require container-loader.inc
>> +
>> +CONTAINER_ENGINE = "docker"
>> +
>> +CONTAINER_ENGINE_PACKAGES ?= "docker.io, apparmor"
>> diff --git a/meta/recipes-support/container-loader/files/container-
>> loader.service.tmpl b/meta/recipes-support/container-
>> loader/files/container-loader.service.tmpl
>> new file mode 100644
>> index 00000000..afde55d3
>> --- /dev/null
>> +++ b/meta/recipes-support/container-loader/files/container-
>> loader.service.tmpl
>> @@ -0,0 +1,11 @@
>> +[Unit]
>> +Description=Load archived container images on boot
>> +After=${CONTAINER_ENGINE}.service
> 
> We need an Requires=${CONTAINER_ENGINE}.service as well. Otherwise just
> the temporal order is defined, but not the logical order.
> 

True.

>> +
>> +[Service]
>> +Type=oneshot
>> +ExecStart=/usr/share/${PN}/container-loader.sh
>> +RemainAfterExit=true
> 
> In case of rootless podman, the user that executes the script matters.
> But I don't know if we want to consider this use-case by now.

Right, thought about that as well but didn't see a generic answer to
that yet.

> 
>> +
>> +[Install]
>> +WantedBy=multi-user.target
>> diff --git a/meta/recipes-support/container-loader/files/container-
>> loader.sh.tmpl b/meta/recipes-support/container-
>> loader/files/container-loader.sh.tmpl
>> new file mode 100755
>> index 00000000..31d27865
>> --- /dev/null
>> +++ b/meta/recipes-support/container-loader/files/container-
>> loader.sh.tmpl
>> @@ -0,0 +1,13 @@
>> +#!/bin/sh
>> +#
>> +# Copyright (c) Siemens AG, 2024
>> +#
>> +# SPDX-License-Identifier: MIT
>> +
>> +set -eu
>> +
>> +while read -r image ref; do
>> +    if [ -z "$(${CONTAINER_ENGINE} images -q "$ref")" ]; then
>> +        ${CONTAINER_ENGINE} load -i /usr/share/${PN}/images/"$image"
> 
> That does not work with all compression formats. Better decompress the
> image manually and pass to stdin of the <tool> load. e.g.

It does work for those we support, see above. And it avoids having to
carry to specific tool also in this script. So, not completely nicer.

> 
> cat $image | unzstd | podman load -
> 
> Also note, that podman needs tons of temporary space to load a
> compressed image (or compressed layers), so you might want to set e.g.
> TMPDIR=/var/tmp as /tmp could be a small memory backed fs.

Strange that open-coding this should actually be worse than letting the
engine to it. Does this hold for oci-archives with compressed layers?

Thanks,
Jan

-- 
Siemens AG, Technology
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/6d1ad398-d4ee-4d9a-9a00-08032d43e2be%40siemens.com.

Re: [RFC][PATCH 2/3] container-loader: Introduce helper to load container images into local registry

['Niedermayr, BENEDIKT' via isar-users]

On Tue, 2024-07-09 at 19:31 +0200, Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> This allows to write dpkg-raw recipes which packages archived container
> images and load them into a local docker or podman registry on boot. The
> scenario behind this is to pre-fill local registries in a way that still
> permits live updates during runtime.
> 
> The loader script only process images which are not yet available under
> the same name and tag in the local registry. Also after loading, the
> archived images stay on the local file system. This allows to perform
> reloading in case the local registry should be emptied (e.g. reset to
> factory state). To reduce the space those original images need, they are
> compressed, by default with xz.
> 
> Separate include files are available to cater the main container
> engines, one for docker and one for podman.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  .../container-loader/container-loader.inc     | 76 +++++++++++++++++++
>  .../container-loader/docker-loader.inc        | 10 +++
>  .../files/container-loader.service.tmpl       | 11 +++
>  .../files/container-loader.sh.tmpl            | 13 ++++
>  .../container-loader/podman-loader.inc        | 10 +++
>  5 files changed, 120 insertions(+)
>  create mode 100644 meta/recipes-support/container-loader/container-loader.inc
>  create mode 100644 meta/recipes-support/container-loader/docker-loader.inc
>  create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl
>  create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl
>  create mode 100644 meta/recipes-support/container-loader/podman-loader.inc
> 
> diff --git a/meta/recipes-support/container-loader/container-loader.inc b/meta/recipes-
> support/container-loader/container-loader.inc
> new file mode 100644
> index 00000000..8e352214
> --- /dev/null
> +++ b/meta/recipes-support/container-loader/container-loader.inc
> @@ -0,0 +1,76 @@
> +# This software is a part of ISAR.
> +# Copyright (c) Siemens AG, 2024
> +#
> +# SPDX-License-Identifier: MIT
> +
> +FILESPATH:append := ":${FILE_DIRNAME}/files"
> +
> +inherit dpkg-raw
> +
> +SRC_URI += " \
> +    file://container-loader.service.tmpl \
> +    file://container-loader.sh.tmpl"
> +
> +TEMPLATE_FILES += " \
> +    container-loader.service.tmpl \
> +    container-loader.sh.tmpl"
> +TEMPLATE_VARS += "CONTAINER_ENGINE"
> +
> +CONTAINER_COMPRESSION ?= "xz"
> +
> +DEBIAN_DEPENDS += " \
> +    ${CONTAINER_ENGINE_PACKAGES} \
> +    ${@', xz-utils' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
> +       ', gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
> +       ''}"
> +
> +CONTAINER_COMPRESSOR = "${@ \
> +    'xz' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
> +    'gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
> +    ''}"
> +
> +python do_install() {
> +    import os
> +
> +    workdir = d.getVar('WORKDIR')
> +    D = d.getVar('D')
> +    PN= d.getVar('PN')
> +
> +    image_list = open(D + "/usr/share/" + PN +"/image.list", "w")
> +
> +    src_uri = d.getVar('SRC_URI').split()
> +    for uri in src_uri:
> +        scheme, host, path, _, _, parm = bb.fetch.decodeurl(uri)
> +        if scheme != "docker":
> +            continue
> +
> +        image_name = host + (path if path != "/" else "")
> +        unpacked_image = workdir + "/" + image_name.replace('/', '.')
> +        dest_dir = D + "/usr/share/" + PN + "/images"
> +        tar_image = dest_dir + "/" + image_name.replace('/', '.') + ".tar"
> +        docker_ref = ":" + parm["tag"] if "tag" in parm else "latest"
> +
> +        cmd = f"skopeo copy dir:{unpacked_image} " \
> +            f"docker-archive:{tar_image}:{image_name}{docker_ref}"
> +        bb.note(f"running: {cmd}")
> +        bb.process.run(cmd)
> +
> +        cmd = f"{d.getVar('CONTAINER_COMPRESSOR')} {tar_image}"
> +        bb.note(f"running: {cmd}")
> +        bb.process.run(cmd)
> +
> +        line = f"{os.path.basename(tar_image)}.{d.getVar('CONTAINER_COMPRESSION')} " + \
> +            image_name + docker_ref
> +        bb.note(f"adding '{line}' to image.list")
> +        image_list.write(line + "\n")
> +
> +    image_list.close()
> +
> +    bb.utils.copyfile(workdir + "/container-loader.sh",
> +                      D + "/usr/share/" + PN + "/container-loader.sh")
> +}
> +do_install[cleandirs] += "${D}/usr/share/${PN}/images"
> +
> +do_prepare_build:append() {
> +    install -v -m 644 ${WORKDIR}/container-loader.service ${S}/debian/${PN}.service
> +}
> diff --git a/meta/recipes-support/container-loader/docker-loader.inc b/meta/recipes-
> support/container-loader/docker-loader.inc
> new file mode 100644
> index 00000000..b864c854
> --- /dev/null
> +++ b/meta/recipes-support/container-loader/docker-loader.inc
> @@ -0,0 +1,10 @@
> +# This software is a part of ISAR.
> +# Copyright (c) Siemens AG, 2024
> +#
> +# SPDX-License-Identifier: MIT
> +
> +require container-loader.inc
> +
> +CONTAINER_ENGINE = "docker"
> +
> +CONTAINER_ENGINE_PACKAGES ?= "docker.io, apparmor"
> diff --git a/meta/recipes-support/container-loader/files/container-loader.service.tmpl
> b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
> new file mode 100644
> index 00000000..afde55d3
> --- /dev/null
> +++ b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
> @@ -0,0 +1,11 @@
> +[Unit]
> +Description=Load archived container images on boot
> +After=${CONTAINER_ENGINE}.service
> +
> +[Service]
> +Type=oneshot
> +ExecStart=/usr/share/${PN}/container-loader.sh
> +RemainAfterExit=true
> +
> +[Install]
> +WantedBy=multi-user.target
> diff --git a/meta/recipes-support/container-loader/files/container-loader.sh.tmpl b/meta/recipes-
> support/container-loader/files/container-loader.sh.tmpl
> new file mode 100755
> index 00000000..31d27865
> --- /dev/null
> +++ b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl
> @@ -0,0 +1,13 @@
> +#!/bin/sh
> +#
> +# Copyright (c) Siemens AG, 2024
> +#
> +# SPDX-License-Identifier: MIT
> +
> +set -eu
> +
> +while read -r image ref; do
> +    if [ -z "$(${CONTAINER_ENGINE} images -q "$ref")" ]; then
> +        ${CONTAINER_ENGINE} load -i /usr/share/${PN}/images/"$image"
> +    fi
> +done < /usr/share/${PN}/image.list

Should the images be deleted after import? I see no reason to preserve them once they have been
added to the local docker engine.
An option to influence that would be good (e.g. CONTAINER_IMAGES_CLEAN).


> diff --git a/meta/recipes-support/container-loader/podman-loader.inc b/meta/recipes-
> support/container-loader/podman-loader.inc
> new file mode 100644
> index 00000000..d2c9a12d
> --- /dev/null
> +++ b/meta/recipes-support/container-loader/podman-loader.inc
> @@ -0,0 +1,10 @@
> +# This software is a part of ISAR.
> +# Copyright (c) Siemens AG, 2024
> +#
> +# SPDX-License-Identifier: MIT
> +
> +require container-loader.inc
> +
> +CONTAINER_ENGINE = "podman"
> +
> +CONTAINER_ENGINE_PACKAGES ?= "podman"

Benedikt

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/ceaf8d24b29fc41b044ca55a7d1f28d8ca8a823e.camel%40siemens.com.

Re: [RFC][PATCH 2/3] container-loader: Introduce helper to load container images into local registry

['Niedermayr, BENEDIKT' via isar-users]

On Tue, 2024-07-09 at 19:31 +0200, Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> This allows to write dpkg-raw recipes which packages archived container
> images and load them into a local docker or podman registry on boot. The
> scenario behind this is to pre-fill local registries in a way that still
> permits live updates during runtime.
> 
> The loader script only process images which are not yet available under
> the same name and tag in the local registry. Also after loading, the
> archived images stay on the local file system. This allows to perform
> reloading in case the local registry should be emptied (e.g. reset to
> factory state). To reduce the space those original images need, they are
> compressed, by default with xz.
> 
> Separate include files are available to cater the main container
> engines, one for docker and one for podman.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  .../container-loader/container-loader.inc     | 76 +++++++++++++++++++
>  .../container-loader/docker-loader.inc        | 10 +++
>  .../files/container-loader.service.tmpl       | 11 +++
>  .../files/container-loader.sh.tmpl            | 13 ++++
>  .../container-loader/podman-loader.inc        | 10 +++
>  5 files changed, 120 insertions(+)
>  create mode 100644 meta/recipes-support/container-loader/container-loader.inc
>  create mode 100644 meta/recipes-support/container-loader/docker-loader.inc
>  create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl
>  create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl
>  create mode 100644 meta/recipes-support/container-loader/podman-loader.inc
> 
> diff --git a/meta/recipes-support/container-loader/container-loader.inc b/meta/recipes-
> support/container-loader/container-loader.inc
> new file mode 100644
> index 00000000..8e352214
> --- /dev/null
> +++ b/meta/recipes-support/container-loader/container-loader.inc
> @@ -0,0 +1,76 @@
> +# This software is a part of ISAR.
> +# Copyright (c) Siemens AG, 2024
> +#
> +# SPDX-License-Identifier: MIT
> +
> +FILESPATH:append := ":${FILE_DIRNAME}/files"
> +
> +inherit dpkg-raw
> +
> +SRC_URI += " \
> +    file://container-loader.service.tmpl \
> +    file://container-loader.sh.tmpl"
> +
> +TEMPLATE_FILES += " \
> +    container-loader.service.tmpl \
> +    container-loader.sh.tmpl"
> +TEMPLATE_VARS += "CONTAINER_ENGINE"
> +
> +CONTAINER_COMPRESSION ?= "xz"
> +
> +DEBIAN_DEPENDS += " \
> +    ${CONTAINER_ENGINE_PACKAGES} \
> +    ${@', xz-utils' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
> +       ', gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
> +       ''}"
> +
> +CONTAINER_COMPRESSOR = "${@ \
> +    'xz' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
> +    'gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
> +    ''}"
> +
> +python do_install() {
> +    import os
> +
> +    workdir = d.getVar('WORKDIR')
> +    D = d.getVar('D')
> +    PN= d.getVar('PN')
> +
> +    image_list = open(D + "/usr/share/" + PN +"/image.list", "w")
> +
> +    src_uri = d.getVar('SRC_URI').split()
> +    for uri in src_uri:
> +        scheme, host, path, _, _, parm = bb.fetch.decodeurl(uri)
> +        if scheme != "docker":
> +            continue
> +
> +        image_name = host + (path if path != "/" else "")
> +        unpacked_image = workdir + "/" + image_name.replace('/', '.')
> +        dest_dir = D + "/usr/share/" + PN + "/images"
> +        tar_image = dest_dir + "/" + image_name.replace('/', '.') + ".tar"
> +        docker_ref = ":" + parm["tag"] if "tag" in parm else "latest"
> +
> +        cmd = f"skopeo copy dir:{unpacked_image} " \
> +            f"docker-archive:{tar_image}:{image_name}{docker_ref}"

regarding performance,
using "xz -T0" or pigz instead of gzip would increase performance and reduce build time a lot.

Benedikt

> +        bb.note(f"running: {cmd}")
> +        bb.process.run(cmd)
> +
> +        cmd = f"{d.getVar('CONTAINER_COMPRESSOR')} {tar_image}"
> +        bb.note(f"running: {cmd}")
> +        bb.process.run(cmd)
> +
> +        line = f"{os.path.basename(tar_image)}.{d.getVar('CONTAINER_COMPRESSION')} " + \
> +            image_name + docker_ref
> +        bb.note(f"adding '{line}' to image.list")
> +        image_list.write(line + "\n")
> +
> +    image_list.close()
> +
> +    bb.utils.copyfile(workdir + "/container-loader.sh",
> +                      D + "/usr/share/" + PN + "/container-loader.sh")
> +}
> +do_install[cleandirs] += "${D}/usr/share/${PN}/images"
> +
> +do_prepare_build:append() {
> +    install -v -m 644 ${WORKDIR}/container-loader.service ${S}/debian/${PN}.service
> +}
> diff --git a/meta/recipes-support/container-loader/docker-loader.inc b/meta/recipes-
> support/container-loader/docker-loader.inc
> new file mode 100644
> index 00000000..b864c854
> --- /dev/null
> +++ b/meta/recipes-support/container-loader/docker-loader.inc
> @@ -0,0 +1,10 @@
> +# This software is a part of ISAR.
> +# Copyright (c) Siemens AG, 2024
> +#
> +# SPDX-License-Identifier: MIT
> +
> +require container-loader.inc
> +
> +CONTAINER_ENGINE = "docker"
> +
> +CONTAINER_ENGINE_PACKAGES ?= "docker.io, apparmor"
> diff --git a/meta/recipes-support/container-loader/files/container-loader.service.tmpl
> b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
> new file mode 100644
> index 00000000..afde55d3
> --- /dev/null
> +++ b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
> @@ -0,0 +1,11 @@
> +[Unit]
> +Description=Load archived container images on boot
> +After=${CONTAINER_ENGINE}.service
> +
> +[Service]
> +Type=oneshot
> +ExecStart=/usr/share/${PN}/container-loader.sh
> +RemainAfterExit=true
> +
> +[Install]
> +WantedBy=multi-user.target
> diff --git a/meta/recipes-support/container-loader/files/container-loader.sh.tmpl b/meta/recipes-
> support/container-loader/files/container-loader.sh.tmpl
> new file mode 100755
> index 00000000..31d27865
> --- /dev/null
> +++ b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl
> @@ -0,0 +1,13 @@
> +#!/bin/sh
> +#
> +# Copyright (c) Siemens AG, 2024
> +#
> +# SPDX-License-Identifier: MIT
> +
> +set -eu
> +
> +while read -r image ref; do
> +    if [ -z "$(${CONTAINER_ENGINE} images -q "$ref")" ]; then
> +        ${CONTAINER_ENGINE} load -i /usr/share/${PN}/images/"$image"
> +    fi
> +done < /usr/share/${PN}/image.list
> diff --git a/meta/recipes-support/container-loader/podman-loader.inc b/meta/recipes-
> support/container-loader/podman-loader.inc
> new file mode 100644
> index 00000000..d2c9a12d
> --- /dev/null
> +++ b/meta/recipes-support/container-loader/podman-loader.inc
> @@ -0,0 +1,10 @@
> +# This software is a part of ISAR.
> +# Copyright (c) Siemens AG, 2024
> +#
> +# SPDX-License-Identifier: MIT
> +
> +require container-loader.inc
> +
> +CONTAINER_ENGINE = "podman"
> +
> +CONTAINER_ENGINE_PACKAGES ?= "podman"

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/e7b2930dfd7ddac9130f998b5e639ab12adccd42.camel%40siemens.com.

Re: [RFC][PATCH 2/3] container-loader: Introduce helper to load container images into local registry

['Jan Kiszka' via isar-users]

On 12.07.24 11:19, Niedermayr, Benedikt (T CED OES-DE) wrote:
> On Tue, 2024-07-09 at 19:31 +0200, Jan Kiszka wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> This allows to write dpkg-raw recipes which packages archived container
>> images and load them into a local docker or podman registry on boot. The
>> scenario behind this is to pre-fill local registries in a way that still
>> permits live updates during runtime.
>>
>> The loader script only process images which are not yet available under
>> the same name and tag in the local registry. Also after loading, the
>> archived images stay on the local file system. This allows to perform
>> reloading in case the local registry should be emptied (e.g. reset to
>> factory state). To reduce the space those original images need, they are
>> compressed, by default with xz.
>>
>> Separate include files are available to cater the main container
>> engines, one for docker and one for podman.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>  .../container-loader/container-loader.inc     | 76 +++++++++++++++++++
>>  .../container-loader/docker-loader.inc        | 10 +++
>>  .../files/container-loader.service.tmpl       | 11 +++
>>  .../files/container-loader.sh.tmpl            | 13 ++++
>>  .../container-loader/podman-loader.inc        | 10 +++
>>  5 files changed, 120 insertions(+)
>>  create mode 100644 meta/recipes-support/container-loader/container-loader.inc
>>  create mode 100644 meta/recipes-support/container-loader/docker-loader.inc
>>  create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl
>>  create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl
>>  create mode 100644 meta/recipes-support/container-loader/podman-loader.inc
>>
>> diff --git a/meta/recipes-support/container-loader/container-loader.inc b/meta/recipes-
>> support/container-loader/container-loader.inc
>> new file mode 100644
>> index 00000000..8e352214
>> --- /dev/null
>> +++ b/meta/recipes-support/container-loader/container-loader.inc
>> @@ -0,0 +1,76 @@
>> +# This software is a part of ISAR.
>> +# Copyright (c) Siemens AG, 2024
>> +#
>> +# SPDX-License-Identifier: MIT
>> +
>> +FILESPATH:append := ":${FILE_DIRNAME}/files"
>> +
>> +inherit dpkg-raw
>> +
>> +SRC_URI += " \
>> +    file://container-loader.service.tmpl \
>> +    file://container-loader.sh.tmpl"
>> +
>> +TEMPLATE_FILES += " \
>> +    container-loader.service.tmpl \
>> +    container-loader.sh.tmpl"
>> +TEMPLATE_VARS += "CONTAINER_ENGINE"
>> +
>> +CONTAINER_COMPRESSION ?= "xz"
>> +
>> +DEBIAN_DEPENDS += " \
>> +    ${CONTAINER_ENGINE_PACKAGES} \
>> +    ${@', xz-utils' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
>> +       ', gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
>> +       ''}"
>> +
>> +CONTAINER_COMPRESSOR = "${@ \
>> +    'xz' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
>> +    'gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
>> +    ''}"
>> +
>> +python do_install() {
>> +    import os
>> +
>> +    workdir = d.getVar('WORKDIR')
>> +    D = d.getVar('D')
>> +    PN= d.getVar('PN')
>> +
>> +    image_list = open(D + "/usr/share/" + PN +"/image.list", "w")
>> +
>> +    src_uri = d.getVar('SRC_URI').split()
>> +    for uri in src_uri:
>> +        scheme, host, path, _, _, parm = bb.fetch.decodeurl(uri)
>> +        if scheme != "docker":
>> +            continue
>> +
>> +        image_name = host + (path if path != "/" else "")
>> +        unpacked_image = workdir + "/" + image_name.replace('/', '.')
>> +        dest_dir = D + "/usr/share/" + PN + "/images"
>> +        tar_image = dest_dir + "/" + image_name.replace('/', '.') + ".tar"
>> +        docker_ref = ":" + parm["tag"] if "tag" in parm else "latest"
>> +
>> +        cmd = f"skopeo copy dir:{unpacked_image} " \
>> +            f"docker-archive:{tar_image}:{image_name}{docker_ref}"
>> +        bb.note(f"running: {cmd}")
>> +        bb.process.run(cmd)
>> +
>> +        cmd = f"{d.getVar('CONTAINER_COMPRESSOR')} {tar_image}"
>> +        bb.note(f"running: {cmd}")
>> +        bb.process.run(cmd)
>> +
>> +        line = f"{os.path.basename(tar_image)}.{d.getVar('CONTAINER_COMPRESSION')} " + \
>> +            image_name + docker_ref
>> +        bb.note(f"adding '{line}' to image.list")
>> +        image_list.write(line + "\n")
>> +
>> +    image_list.close()
>> +
>> +    bb.utils.copyfile(workdir + "/container-loader.sh",
>> +                      D + "/usr/share/" + PN + "/container-loader.sh")
>> +}
>> +do_install[cleandirs] += "${D}/usr/share/${PN}/images"
>> +
>> +do_prepare_build:append() {
>> +    install -v -m 644 ${WORKDIR}/container-loader.service ${S}/debian/${PN}.service
>> +}
>> diff --git a/meta/recipes-support/container-loader/docker-loader.inc b/meta/recipes-
>> support/container-loader/docker-loader.inc
>> new file mode 100644
>> index 00000000..b864c854
>> --- /dev/null
>> +++ b/meta/recipes-support/container-loader/docker-loader.inc
>> @@ -0,0 +1,10 @@
>> +# This software is a part of ISAR.
>> +# Copyright (c) Siemens AG, 2024
>> +#
>> +# SPDX-License-Identifier: MIT
>> +
>> +require container-loader.inc
>> +
>> +CONTAINER_ENGINE = "docker"
>> +
>> +CONTAINER_ENGINE_PACKAGES ?= "docker.io, apparmor"
>> diff --git a/meta/recipes-support/container-loader/files/container-loader.service.tmpl
>> b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
>> new file mode 100644
>> index 00000000..afde55d3
>> --- /dev/null
>> +++ b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
>> @@ -0,0 +1,11 @@
>> +[Unit]
>> +Description=Load archived container images on boot
>> +After=${CONTAINER_ENGINE}.service
>> +
>> +[Service]
>> +Type=oneshot
>> +ExecStart=/usr/share/${PN}/container-loader.sh
>> +RemainAfterExit=true
>> +
>> +[Install]
>> +WantedBy=multi-user.target
>> diff --git a/meta/recipes-support/container-loader/files/container-loader.sh.tmpl b/meta/recipes-
>> support/container-loader/files/container-loader.sh.tmpl
>> new file mode 100755
>> index 00000000..31d27865
>> --- /dev/null
>> +++ b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl
>> @@ -0,0 +1,13 @@
>> +#!/bin/sh
>> +#
>> +# Copyright (c) Siemens AG, 2024
>> +#
>> +# SPDX-License-Identifier: MIT
>> +
>> +set -eu
>> +
>> +while read -r image ref; do
>> +    if [ -z "$(${CONTAINER_ENGINE} images -q "$ref")" ]; then
>> +        ${CONTAINER_ENGINE} load -i /usr/share/${PN}/images/"$image"
>> +    fi
>> +done < /usr/share/${PN}/image.list
> 
> Should the images be deleted after import? I see no reason to preserve them once they have been
> added to the local docker engine.

You can't delete the images if they are part of a r/o rootfs.
Furthermore, you may want to keep them for the purpose of factory resets.

> An option to influence that would be good (e.g. CONTAINER_IMAGES_CLEAN).
> 

Something like this is what I was thinking about already.

Jan

> 
>> diff --git a/meta/recipes-support/container-loader/podman-loader.inc b/meta/recipes-
>> support/container-loader/podman-loader.inc
>> new file mode 100644
>> index 00000000..d2c9a12d
>> --- /dev/null
>> +++ b/meta/recipes-support/container-loader/podman-loader.inc
>> @@ -0,0 +1,10 @@
>> +# This software is a part of ISAR.
>> +# Copyright (c) Siemens AG, 2024
>> +#
>> +# SPDX-License-Identifier: MIT
>> +
>> +require container-loader.inc
>> +
>> +CONTAINER_ENGINE = "podman"
>> +
>> +CONTAINER_ENGINE_PACKAGES ?= "podman"
> 
> Benedikt

-- 
Siemens AG, Technology
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/960c2ec4-c365-4aff-998b-569207354dcb%40siemens.com.

Re: [RFC][PATCH 2/3] container-loader: Introduce helper to load container images into local registry

['Jan Kiszka' via isar-users]

On 12.07.24 14:11, Niedermayr, Benedikt (T CED OES-DE) wrote:
> On Tue, 2024-07-09 at 19:31 +0200, Jan Kiszka wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> This allows to write dpkg-raw recipes which packages archived container
>> images and load them into a local docker or podman registry on boot. The
>> scenario behind this is to pre-fill local registries in a way that still
>> permits live updates during runtime.
>>
>> The loader script only process images which are not yet available under
>> the same name and tag in the local registry. Also after loading, the
>> archived images stay on the local file system. This allows to perform
>> reloading in case the local registry should be emptied (e.g. reset to
>> factory state). To reduce the space those original images need, they are
>> compressed, by default with xz.
>>
>> Separate include files are available to cater the main container
>> engines, one for docker and one for podman.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>  .../container-loader/container-loader.inc     | 76 +++++++++++++++++++
>>  .../container-loader/docker-loader.inc        | 10 +++
>>  .../files/container-loader.service.tmpl       | 11 +++
>>  .../files/container-loader.sh.tmpl            | 13 ++++
>>  .../container-loader/podman-loader.inc        | 10 +++
>>  5 files changed, 120 insertions(+)
>>  create mode 100644 meta/recipes-support/container-loader/container-loader.inc
>>  create mode 100644 meta/recipes-support/container-loader/docker-loader.inc
>>  create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl
>>  create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl
>>  create mode 100644 meta/recipes-support/container-loader/podman-loader.inc
>>
>> diff --git a/meta/recipes-support/container-loader/container-loader.inc b/meta/recipes-
>> support/container-loader/container-loader.inc
>> new file mode 100644
>> index 00000000..8e352214
>> --- /dev/null
>> +++ b/meta/recipes-support/container-loader/container-loader.inc
>> @@ -0,0 +1,76 @@
>> +# This software is a part of ISAR.
>> +# Copyright (c) Siemens AG, 2024
>> +#
>> +# SPDX-License-Identifier: MIT
>> +
>> +FILESPATH:append := ":${FILE_DIRNAME}/files"
>> +
>> +inherit dpkg-raw
>> +
>> +SRC_URI += " \
>> +    file://container-loader.service.tmpl \
>> +    file://container-loader.sh.tmpl"
>> +
>> +TEMPLATE_FILES += " \
>> +    container-loader.service.tmpl \
>> +    container-loader.sh.tmpl"
>> +TEMPLATE_VARS += "CONTAINER_ENGINE"
>> +
>> +CONTAINER_COMPRESSION ?= "xz"
>> +
>> +DEBIAN_DEPENDS += " \
>> +    ${CONTAINER_ENGINE_PACKAGES} \
>> +    ${@', xz-utils' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
>> +       ', gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
>> +       ''}"
>> +
>> +CONTAINER_COMPRESSOR = "${@ \
>> +    'xz' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
>> +    'gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
>> +    ''}"
>> +
>> +python do_install() {
>> +    import os
>> +
>> +    workdir = d.getVar('WORKDIR')
>> +    D = d.getVar('D')
>> +    PN= d.getVar('PN')
>> +
>> +    image_list = open(D + "/usr/share/" + PN +"/image.list", "w")
>> +
>> +    src_uri = d.getVar('SRC_URI').split()
>> +    for uri in src_uri:
>> +        scheme, host, path, _, _, parm = bb.fetch.decodeurl(uri)
>> +        if scheme != "docker":
>> +            continue
>> +
>> +        image_name = host + (path if path != "/" else "")
>> +        unpacked_image = workdir + "/" + image_name.replace('/', '.')
>> +        dest_dir = D + "/usr/share/" + PN + "/images"
>> +        tar_image = dest_dir + "/" + image_name.replace('/', '.') + ".tar"
>> +        docker_ref = ":" + parm["tag"] if "tag" in parm else "latest"
>> +
>> +        cmd = f"skopeo copy dir:{unpacked_image} " \
>> +            f"docker-archive:{tar_image}:{image_name}{docker_ref}"
> 
> regarding performance,
> using "xz -T0" or pigz instead of gzip would increase performance and reduce build time a lot.
> 

(looks like you wanted to comment below)

> Benedikt
> 
>> +        bb.note(f"running: {cmd}")
>> +        bb.process.run(cmd)
>> +
>> +        cmd = f"{d.getVar('CONTAINER_COMPRESSOR')} {tar_image}"

We can tune CONTAINER_COMPRESSOR and its dependencies. I guess I will
also follow Felix suggestion to open-code decompression so that we avoid
shortcomings of podman's built-in decompression.

Jan

>> +        bb.note(f"running: {cmd}")
>> +        bb.process.run(cmd)
>> +
>> +        line = f"{os.path.basename(tar_image)}.{d.getVar('CONTAINER_COMPRESSION')} " + \
>> +            image_name + docker_ref
>> +        bb.note(f"adding '{line}' to image.list")
>> +        image_list.write(line + "\n")
>> +
>> +    image_list.close()
>> +
>> +    bb.utils.copyfile(workdir + "/container-loader.sh",
>> +                      D + "/usr/share/" + PN + "/container-loader.sh")
>> +}
>> +do_install[cleandirs] += "${D}/usr/share/${PN}/images"
>> +
>> +do_prepare_build:append() {
>> +    install -v -m 644 ${WORKDIR}/container-loader.service ${S}/debian/${PN}.service
>> +}
>> diff --git a/meta/recipes-support/container-loader/docker-loader.inc b/meta/recipes-
>> support/container-loader/docker-loader.inc
>> new file mode 100644
>> index 00000000..b864c854
>> --- /dev/null
>> +++ b/meta/recipes-support/container-loader/docker-loader.inc
>> @@ -0,0 +1,10 @@
>> +# This software is a part of ISAR.
>> +# Copyright (c) Siemens AG, 2024
>> +#
>> +# SPDX-License-Identifier: MIT
>> +
>> +require container-loader.inc
>> +
>> +CONTAINER_ENGINE = "docker"
>> +
>> +CONTAINER_ENGINE_PACKAGES ?= "docker.io, apparmor"
>> diff --git a/meta/recipes-support/container-loader/files/container-loader.service.tmpl
>> b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
>> new file mode 100644
>> index 00000000..afde55d3
>> --- /dev/null
>> +++ b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
>> @@ -0,0 +1,11 @@
>> +[Unit]
>> +Description=Load archived container images on boot
>> +After=${CONTAINER_ENGINE}.service
>> +
>> +[Service]
>> +Type=oneshot
>> +ExecStart=/usr/share/${PN}/container-loader.sh
>> +RemainAfterExit=true
>> +
>> +[Install]
>> +WantedBy=multi-user.target
>> diff --git a/meta/recipes-support/container-loader/files/container-loader.sh.tmpl b/meta/recipes-
>> support/container-loader/files/container-loader.sh.tmpl
>> new file mode 100755
>> index 00000000..31d27865
>> --- /dev/null
>> +++ b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl
>> @@ -0,0 +1,13 @@
>> +#!/bin/sh
>> +#
>> +# Copyright (c) Siemens AG, 2024
>> +#
>> +# SPDX-License-Identifier: MIT
>> +
>> +set -eu
>> +
>> +while read -r image ref; do
>> +    if [ -z "$(${CONTAINER_ENGINE} images -q "$ref")" ]; then
>> +        ${CONTAINER_ENGINE} load -i /usr/share/${PN}/images/"$image"
>> +    fi
>> +done < /usr/share/${PN}/image.list
>> diff --git a/meta/recipes-support/container-loader/podman-loader.inc b/meta/recipes-
>> support/container-loader/podman-loader.inc
>> new file mode 100644
>> index 00000000..d2c9a12d
>> --- /dev/null
>> +++ b/meta/recipes-support/container-loader/podman-loader.inc
>> @@ -0,0 +1,10 @@
>> +# This software is a part of ISAR.
>> +# Copyright (c) Siemens AG, 2024
>> +#
>> +# SPDX-License-Identifier: MIT
>> +
>> +require container-loader.inc
>> +
>> +CONTAINER_ENGINE = "podman"
>> +
>> +CONTAINER_ENGINE_PACKAGES ?= "podman"
> 

-- 
Siemens AG, Technology
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/919c79bd-c444-424c-9a85-627b444bd9b3%40siemens.com.