[PATCH v2 0/5] Introduce container fetcher and pre-loader

[PATCH v2 0/5] Introduce container fetcher and pre-loader

['Jan Kiszka' via isar-users]

We noticed internally that there are at least 3 implementations for
solving the task of pre-loading container images into isar-built base
systems, may it be that the images are not publicly available or the
device is only poorly connected, if at all. This series tries to solve
the task generically - hopefully.

Changes in v2:
 - switch to open-coded unpack before loading
 - switch to zstd as default compression
 - use default compression settings from bitbake.conf
 - add support for delete-after-load (opt-in)
 - factor out fetched container installation into separate task
   (should make self-built container packaging easier)
 - use BPN instead of PN
 - add "Requires" to systemd service
 - use PACKAGE_ARCH to select container arch (rather than DISTRO_ARCH)
 - add CI test
 - add documentation

Jan

Jan Kiszka (5):
  Introduce fetcher from container registries
  container-loader: Introduce helper to load container images into local
    registry
  meta-isar: Add demo packages for installing prebuilt containers
  ci: Add test cases for container fetching and loading
  doc: Describe how to use the container fetcher and loader

 doc/user_manual.md                            | 60 ++++++++++++
 kas/package/Kconfig                           | 19 ++++
 kas/package/pkg_prebuilt-docker-img.yaml      |  9 ++
 kas/package/pkg_prebuilt-podman-img.yaml      |  9 ++
 .../prebuilt-docker-img_0.1.bb                | 12 +++
 .../prebuilt-podman-img_0.1.bb                | 10 ++
 .../recipes-core/images/isar-image-ci.bb      |  2 +
 meta/classes/dpkg-base.bbclass                |  6 ++
 meta/lib/container_fetcher.py                 | 90 ++++++++++++++++++
 .../container-loader/container-loader.inc     | 94 +++++++++++++++++++
 .../container-loader/docker-loader.inc        | 10 ++
 .../files/container-loader.service.tmpl       | 12 +++
 .../files/container-loader.sh.tmpl            | 18 ++++
 .../container-loader/podman-loader.inc        | 10 ++
 testsuite/citest.py                           | 21 +++++
 15 files changed, 382 insertions(+)
 create mode 100644 kas/package/pkg_prebuilt-docker-img.yaml
 create mode 100644 kas/package/pkg_prebuilt-podman-img.yaml
 create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
 create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb
 create mode 100644 meta/lib/container_fetcher.py
 create mode 100644 meta/recipes-support/container-loader/container-loader.inc
 create mode 100644 meta/recipes-support/container-loader/docker-loader.inc
 create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl
 create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl
 create mode 100644 meta/recipes-support/container-loader/podman-loader.inc

-- 
2.43.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/cover.1721038111.git.jan.kiszka%40siemens.com.

[PATCH v2 1/5] Introduce fetcher from container registries

['Jan Kiszka' via isar-users]

From: Jan Kiszka <jan.kiszka@siemens.com>

This bitbake fetcher allows to pull container images from registries,
store them in the download cache and transfer them into the workdir of
recipes requesting the image. The format of the URL is

docker://[<host>/]<image>;digest=sha256:...[;tag=<tag>]

Fetching without digest is supported but will cause a warning, just like
downloading via wget without a checksum. If tag is left out, "latest" is
used.

The fetcher will try to pull all available variants of a multi-arch
image. If this is not needed, you can also directly specify the image
digest of a specific architecture.

Future versions may also introduce full unpacking of the fetched
container layers in workdir if use cases come up.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg-base.bbclass |  6 +++
 meta/lib/container_fetcher.py  | 90 ++++++++++++++++++++++++++++++++++
 2 files changed, 96 insertions(+)
 create mode 100644 meta/lib/container_fetcher.py

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 789d6c74..d90b32a9 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -98,6 +98,12 @@ python() {
     if len(d.getVar('SRC_APT').strip()) > 0:
         bb.build.addtask('apt_unpack', 'do_patch', '', d)
         bb.build.addtask('cleanall_apt', 'do_cleanall', '', d)
+
+    # container docker fetcher
+    import container_fetcher
+    from bb.fetch2 import methods
+
+    methods.append(container_fetcher.Container())
 }
 
 do_apt_fetch() {
diff --git a/meta/lib/container_fetcher.py b/meta/lib/container_fetcher.py
new file mode 100644
index 00000000..8513e246
--- /dev/null
+++ b/meta/lib/container_fetcher.py
@@ -0,0 +1,90 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+import os
+import re
+from   bb.fetch2 import FetchMethod
+from   bb.fetch2 import logger
+from   bb.fetch2 import MissingChecksumEvent
+from   bb.fetch2 import NoChecksumError
+from   bb.fetch2 import runfetchcmd
+from   bb.progress import LineFilterProgressHandler
+
+class SkopeoProgressHandler(LineFilterProgressHandler):
+    def __init__(self, d):
+        super(SkopeoProgressHandler, self).__init__(d)
+        self._fire_progress(0)
+
+    def writeline(self, line):
+        match = re.findall(r'^Copying image .*\(([0-9]+/[0-9]+)\)$', line)
+        if match:
+            state = match[0].split('/')
+            progress = (int(state[0]) * 100) / int(state[1])
+            self.update(progress)
+        return True
+
+
+class Container(FetchMethod):
+    def supports(self, ud, d):
+        return ud.type in ['docker']
+
+    def urldata_init(self, ud, d):
+        ud.tag = "latest"
+        if "tag" in ud.parm:
+            ud.tag = ud.parm["tag"]
+
+        ud.digest = None
+        if "digest" in ud.parm:
+            ud.digest = ud.parm["digest"]
+
+        container_name = ud.host + (ud.path if ud.path != "/" else "")
+        ud.container_src = container_name + \
+            ("@" + ud.digest if ud.digest else ":" + ud.tag)
+        ud.localname = container_name.replace('/', '.')
+        ud.localfile = "container-images/" + ud.localname + \
+            "_" + (ud.digest.replace(":", "-") if ud.digest else ud.tag)
+
+    def download(self, ud, d):
+        progresshandler = SkopeoProgressHandler(d)
+        runfetchcmd(f"skopeo copy --preserve-digests --all docker://{ud.container_src} dir:{ud.localfile}",
+                    d, log=progresshandler)
+
+        if ud.digest:
+            return
+
+        checksum = bb.utils.sha256_file(ud.localpath + "/manifest.json")
+        checksum_line = f"SRC_URI = \"{ud.url};digest=sha256:{checksum}\""
+
+        strict = d.getVar("BB_STRICT_CHECKSUM") or "0"
+
+        # If strict checking enabled and neither sum defined, raise error
+        if strict == "1":
+            raise NoChecksumError(checksum_line)
+
+        checksum_event = {"sha256sum": checksum}
+        bb.event.fire(MissingChecksumEvent(ud.url, **checksum_event), d)
+
+        if strict == "ignore":
+            return
+
+        # Log missing digest so user can more easily add it
+        logger.warning(
+            f"Missing checksum for '{ud.localpath}', consider using this " \
+            f"SRC_URI in the recipe:\n{checksum_line}")
+
+    def unpack(self, ud, rootdir, d):
+        arch = d.getVar('PACKAGE_ARCH')
+        variant_opt = ""
+        if arch == "armhf":
+            arch = "arm"
+            variant_opt = "--override-variant v7"
+        elif arch == "armel":
+            arch = "arm"
+            variant_opt = "--override-variant v6"
+        runfetchcmd(f"skopeo --override-arch {arch} {variant_opt} " \
+            f"copy dir:{ud.localpath} dir:{rootdir + '/' + ud.localname}", d)
+
+    def clean(self, ud, d):
+        bb.utils.remove(ud.localpath, recurse=True)
-- 
2.43.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/b0493bec83353990199120a5fcf43676e995bba8.1721038111.git.jan.kiszka%40siemens.com.

[PATCH v2 2/5] container-loader: Introduce helper to load container images into local registry

['Jan Kiszka' via isar-users]

From: Jan Kiszka <jan.kiszka@siemens.com>

This allows to write dpkg-raw recipes which packages archived container
images and load them into a local docker or podman registry on boot. The
scenario behind this is to pre-fill local registries in a way that still
permits live updates during runtime.

The loader script only process images which are not yet available under
the same name and tag in the local registry. Also after loading, the
archived images stay on the local file system. This allows to perform
reloading in case the local registry should be emptied (e.g. reset to
factory state). To reduce the space those original images need, they are
compressed, by default with zstd.

Separate include files are available to cater the main container
engines, one for docker and one for podman.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 .../container-loader/container-loader.inc     | 94 +++++++++++++++++++
 .../container-loader/docker-loader.inc        | 10 ++
 .../files/container-loader.service.tmpl       | 12 +++
 .../files/container-loader.sh.tmpl            | 18 ++++
 .../container-loader/podman-loader.inc        | 10 ++
 5 files changed, 144 insertions(+)
 create mode 100644 meta/recipes-support/container-loader/container-loader.inc
 create mode 100644 meta/recipes-support/container-loader/docker-loader.inc
 create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl
 create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl
 create mode 100644 meta/recipes-support/container-loader/podman-loader.inc

diff --git a/meta/recipes-support/container-loader/container-loader.inc b/meta/recipes-support/container-loader/container-loader.inc
new file mode 100644
index 00000000..a0c2ddb3
--- /dev/null
+++ b/meta/recipes-support/container-loader/container-loader.inc
@@ -0,0 +1,94 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+FILESPATH:append := ":${FILE_DIRNAME}/files"
+
+inherit dpkg-raw
+
+SRC_URI += " \
+    file://container-loader.service.tmpl \
+    file://container-loader.sh.tmpl"
+
+CONTAINER_COMPRESSION ?= "zst"
+CONTAINER_DELETE_AFTER_LOAD ?= "0"
+
+DEBIAN_DEPENDS += " \
+    ${CONTAINER_ENGINE_PACKAGES} \
+    ${@', gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
+       ', zstd' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \
+       ', xz-utils' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
+       ''}"
+
+CONTAINER_COMPRESSOR_CMD = "${@ \
+    'gzip -f -9 -n --rsyncable' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
+    'xz -f ${XZ_DEFAULTS}' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
+    'zstd -f --rm ${ZSTD_DEFAULTS}' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \
+    ''}"
+
+CONTAINER_DECOMPRESSOR_CMD = "${@ \
+    'gzip -c -d -n' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
+    'xz -c -d -T0' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
+    'pzstd -c -d' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \
+    ''}"
+
+TEMPLATE_FILES += " \
+    container-loader.service.tmpl \
+    container-loader.sh.tmpl"
+TEMPLATE_VARS += " \
+    CONTAINER_ENGINE \
+    CONTAINER_DECOMPRESSOR_CMD \
+    CONTAINER_DELETE_AFTER_LOAD"
+
+do_install() {
+    install -m 755 ${WORKDIR}/container-loader.sh ${D}/usr/share/${BPN}
+}
+do_install[cleandirs] += " \
+    ${D}/usr/share/${BPN} \
+    ${D}/usr/share/${BPN}/images"
+
+python do_install_fetched_containers() {
+    import os
+
+    workdir = d.getVar('WORKDIR')
+    D = d.getVar('D')
+    BPN = d.getVar('BPN')
+
+    image_list = open(D + "/usr/share/" + BPN + "/image.list", "w")
+
+    src_uri = d.getVar('SRC_URI').split()
+    for uri in src_uri:
+        scheme, host, path, _, _, parm = bb.fetch.decodeurl(uri)
+        if scheme != "docker":
+            continue
+
+        image_name = host + (path if path != "/" else "")
+        unpacked_image = workdir + "/" + image_name.replace('/', '.')
+        dest_dir = D + "/usr/share/" + BPN + "/images"
+        tar_image = dest_dir + "/" + image_name.replace('/', '.') + ".tar"
+        docker_ref = ":" + parm["tag"] if "tag" in parm else "latest"
+
+        bb.utils.remove(tar_image)
+        cmd = f"skopeo copy dir:{unpacked_image} " \
+            f"docker-archive:{tar_image}:{image_name}{docker_ref}"
+        bb.note(f"running: {cmd}")
+        bb.process.run(cmd)
+
+        cmd = f"{d.getVar('CONTAINER_COMPRESSOR_CMD')} {tar_image}"
+        bb.note(f"running: {cmd}")
+        bb.process.run(cmd)
+
+        line = f"{os.path.basename(tar_image)}.{d.getVar('CONTAINER_COMPRESSION')} " + \
+            image_name + docker_ref
+        bb.note(f"adding '{line}' to image.list")
+        image_list.write(line + "\n")
+
+    image_list.close()
+}
+
+addtask install_fetched_containers after do_install before do_prepare_build
+
+do_prepare_build:append() {
+    install -v -m 644 ${WORKDIR}/container-loader.service ${S}/debian/${BPN}.service
+}
diff --git a/meta/recipes-support/container-loader/docker-loader.inc b/meta/recipes-support/container-loader/docker-loader.inc
new file mode 100644
index 00000000..b864c854
--- /dev/null
+++ b/meta/recipes-support/container-loader/docker-loader.inc
@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+require container-loader.inc
+
+CONTAINER_ENGINE = "docker"
+
+CONTAINER_ENGINE_PACKAGES ?= "docker.io, apparmor"
diff --git a/meta/recipes-support/container-loader/files/container-loader.service.tmpl b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
new file mode 100644
index 00000000..1638eaf2
--- /dev/null
+++ b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
@@ -0,0 +1,12 @@
+[Unit]
+Description=Load archived container images on boot
+After=${CONTAINER_ENGINE}.service
+Requires=${CONTAINER_ENGINE}.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/share/${BPN}/container-loader.sh
+RemainAfterExit=true
+
+[Install]
+WantedBy=multi-user.target
diff --git a/meta/recipes-support/container-loader/files/container-loader.sh.tmpl b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl
new file mode 100755
index 00000000..b6abec92
--- /dev/null
+++ b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl
@@ -0,0 +1,18 @@
+#!/bin/sh
+#
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+set -eu
+
+while read -r image ref; do
+    if [ -e /usr/share/${BPN}/images/"$image" ] && \
+       [ -z "$(${CONTAINER_ENGINE} images -q "$ref")" ]; then
+        ${CONTAINER_DECOMPRESSOR_CMD} /usr/share/${BPN}/images/"$image" | \
+            ${CONTAINER_ENGINE} load
+        if [ "${CONTAINER_DELETE_AFTER_LOAD}" = "1" ]; then
+            rm -f /usr/share/${BPN}/images/"$image"
+        fi
+    fi
+done < /usr/share/${BPN}/image.list
diff --git a/meta/recipes-support/container-loader/podman-loader.inc b/meta/recipes-support/container-loader/podman-loader.inc
new file mode 100644
index 00000000..d2c9a12d
--- /dev/null
+++ b/meta/recipes-support/container-loader/podman-loader.inc
@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+require container-loader.inc
+
+CONTAINER_ENGINE = "podman"
+
+CONTAINER_ENGINE_PACKAGES ?= "podman"
-- 
2.43.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/27306a537db0da8d51a0ca709b9113248331d340.1721038111.git.jan.kiszka%40siemens.com.

[PATCH v2 3/5] meta-isar: Add demo packages for installing prebuilt containers

['Jan Kiszka' via isar-users]

From: Jan Kiszka <jan.kiszka@siemens.com>

One recipe for docker, one for podman. Both pull from a registry that,
in contrast to infamous dockerhub, should not throttle CI jobs running
these frequently for testing purposes.

The podman variant of the recipe is intentionally leaving out the digest
to trigger the related warning of the container fetcher.

These demos also come with kas integration.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 kas/package/Kconfig                           | 19 +++++++++++++++++++
 kas/package/pkg_prebuilt-docker-img.yaml      |  9 +++++++++
 kas/package/pkg_prebuilt-podman-img.yaml      |  9 +++++++++
 .../prebuilt-docker-img_0.1.bb                | 12 ++++++++++++
 .../prebuilt-podman-img_0.1.bb                | 10 ++++++++++
 5 files changed, 59 insertions(+)
 create mode 100644 kas/package/pkg_prebuilt-docker-img.yaml
 create mode 100644 kas/package/pkg_prebuilt-podman-img.yaml
 create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
 create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb

diff --git a/kas/package/Kconfig b/kas/package/Kconfig
index 35ba7cf1..395c3a08 100644
--- a/kas/package/Kconfig
+++ b/kas/package/Kconfig
@@ -146,6 +146,25 @@ config KAS_INCLUDE_PACKAGE_ISAR_CI_SSH_SETUP
         default "kas/package/pkg_sshd-regen-keys.yaml"
         depends on PACKAGE_ISAR_CI_SSH_SETUP
 
+
+config PACKAGE_PREBUILT_DOCKER_IMG
+        bool "prebuilt-docker-img"
+        default y
+
+config KAS_INCLUDE_PACKAGE_PREBUILT_DOCKER_IMG
+        string
+        default "kas/package/pkg_prebuilt-docker-img.yaml"
+        depends on PACKAGE_PREBUILT_DOCKER_IMG
+
+config PACKAGE_PREBUILT_PODMAN_IMG
+        bool "prebuilt-podman-img"
+        default y
+
+config KAS_INCLUDE_PACKAGE_PREBUILT_PODMAN_IMG
+        string
+        default "kas/package/pkg_prebuilt-podman-img.yaml"
+        depends on PACKAGE_PREBUILT_PODMAN_IMG
+
 endmenu
 
 config KAS_IMAGE_PREINSTALL
diff --git a/kas/package/pkg_prebuilt-docker-img.yaml b/kas/package/pkg_prebuilt-docker-img.yaml
new file mode 100644
index 00000000..df96a484
--- /dev/null
+++ b/kas/package/pkg_prebuilt-docker-img.yaml
@@ -0,0 +1,9 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+
+header:
+  version: 14
+
+local_conf_header:
+  package-prebuilt-docker-img: |
+    IMAGE_INSTALL:append = " prebuilt-docker-img"
diff --git a/kas/package/pkg_prebuilt-podman-img.yaml b/kas/package/pkg_prebuilt-podman-img.yaml
new file mode 100644
index 00000000..d0b8da1c
--- /dev/null
+++ b/kas/package/pkg_prebuilt-podman-img.yaml
@@ -0,0 +1,9 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+
+header:
+  version: 14
+
+local_conf_header:
+  package-prebuilt-podman-img: |
+    IMAGE_INSTALL:append = " prebuilt-podman-img"
diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
new file mode 100644
index 00000000..0dfc9b8f
--- /dev/null
+++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
@@ -0,0 +1,12 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+require recipes-support/container-loader/docker-loader.inc
+
+CONTAINER_DELETE_AFTER_LOAD = "1"
+
+SRC_URI += "\
+    docker://quay.io/libpod/alpine;digest=sha256:fa93b01658e3a5a1686dc3ae55f170d8de487006fb53a28efcd12ab0710a2e5f;tag=3.10.2 \
+    "
diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb
new file mode 100644
index 00000000..e671a494
--- /dev/null
+++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb
@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+require recipes-support/container-loader/podman-loader.inc
+
+SRC_URI += "\
+    docker://quay.io/libpod/alpine;tag=latest \
+    "
-- 
2.43.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/fc4a492ac8a22d0768ff9d967f940caa4c546b18.1721038111.git.jan.kiszka%40siemens.com.

[PATCH v2 4/5] ci: Add test cases for container fetching and loading

['Jan Kiszka' via isar-users]

From: Jan Kiszka <jan.kiszka@siemens.com>

This plugs the two example recipes for loading container images into
VM-based testing. The test consists of running 'true' in the installed
alpine images.

Rather than enabling the ci user to do password-less sudo, this uses su
with the piped-in password. Another trick needed is to poll for the
images because loading is performed asynchronously.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 .../recipes-core/images/isar-image-ci.bb      |  2 ++
 testsuite/citest.py                           | 21 +++++++++++++++++++
 2 files changed, 23 insertions(+)

diff --git a/meta-test/recipes-core/images/isar-image-ci.bb b/meta-test/recipes-core/images/isar-image-ci.bb
index e5d51e6e..9133da74 100644
--- a/meta-test/recipes-core/images/isar-image-ci.bb
+++ b/meta-test/recipes-core/images/isar-image-ci.bb
@@ -16,6 +16,7 @@ IMAGE_INSTALL += "sshd-regen-keys"
 
 # qemuamd64-bookworm
 WKS_FILE:qemuamd64:debian-bookworm ?= "multipart-efi.wks"
+IMAGE_INSTALL:append:qemuamd64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img"
 
 # qemuamd64-bullseye
 IMAGE_FSTYPES:append:qemuamd64:debian-bullseye ?= " cpio.gz tar.gz"
@@ -51,3 +52,4 @@ IMAGER_INSTALL:append:qemuarm:debian-bookworm ?= " ${SYSTEMD_BOOTLOADER_INSTALL}
 # qemuarm64-bookworm
 IMAGE_FSTYPES:append:qemuarm64:debian-bookworm ?= " wic.xz"
 IMAGER_INSTALL:append:qemuarm64:debian-bookworm ?= " ${GRUB_BOOTLOADER_INSTALL}"
+IMAGE_INSTALL:append:qemuarm64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img"
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 8dd907d0..539c9440 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -522,3 +522,24 @@ class VmBootTestFull(CIBaseTest):
         self.init()
         self.vm_start('mipsel','bookworm', image='isar-image-ci',
                       script='test_kernel_module.sh example_module')
+
+
+    def test_amd64_bookworm_prebuilt_containers(self):
+        self.init()
+        self.vm_start('amd64', 'bookworm', image='isar-image-ci',
+                      cmd='echo root | su -c \'' \
+                          'PATH=\$PATH:/usr/sbin;' \
+                          'for n in \$(seq 30); do docker images | grep -q alpine && break; sleep 10; done;' \
+                          'docker run --rm quay.io/libpod/alpine:3.10.2 true && ' \
+                          'for n in \$(seq 30); do podman images | grep -q alpine && break; sleep 10; done;' \
+                          'podman run --rm quay.io/libpod/alpine:latest true\'')
+
+    def test_arm64_bookworm_prebuilt_containers(self):
+        self.init()
+        self.vm_start('arm64', 'bookworm', image='isar-image-ci',
+                      cmd='echo root | su -c \'' \
+                          'PATH=\$PATH:/usr/sbin;' \
+                          'for n in \$(seq 30); do docker images | grep -q alpine && break; sleep 10; done;' \
+                          'docker run --rm quay.io/libpod/alpine:3.10.2 true && ' \
+                          'for n in \$(seq 30); do podman images | grep -q alpine && break; sleep 10; done;' \
+                          'podman run --rm quay.io/libpod/alpine:latest true\'')
-- 
2.43.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/67bcaf0f3ff3d5939f7df7061384758391e92443.1721038111.git.jan.kiszka%40siemens.com.

[PATCH v2 5/5] doc: Describe how to use the container fetcher and loader

['Jan Kiszka' via isar-users]

From: Jan Kiszka <jan.kiszka@siemens.com>

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 doc/user_manual.md | 60 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 60 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 776ae52c..733b2b30 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -1519,3 +1519,63 @@ SBUILD_CHROOT_PREINSTALL_EXTRA += "<base packages>"
 
 Then, in the dpkg recipe of your package, simply set `SBUILD_FLAVOR = "<your flavor>"`.
 To install additional packages into the sbuild chroot, add them to `SBUILD_CHROOT_PREINSTALL_EXTRA`.
+
+## Pre-install container images
+
+If an isar-generated image shall provides a container runtime, it may also be
+desirable to pre-install container images to avoid having to download them on
+first boot or because they may not be accessible outside of the build
+environment. Isar supports this scenario via two services, a container fetcher
+and a container loader.
+
+### Bitbake fetcher for containers
+
+The bitbake fetching protocol "docker://" allows to download pre-built images
+from container registries. The URL consists of the image path, followed by
+a recommened digests in the form `digest=sha256:<sha256sum>` and an optional
+tag in the form `tag=<tag>`. A digest is preferred over a tag to identify the
+an when fetching as it also allows to validate its integrity. If tag is not
+specified, `latest` is used as tag name.
+
+When specifying a multi-arch image, the fetcher will download the images for
+all available architectures. If this is not desired, directly specify the
+digest of the desired architecture manifest instead of that the manifest list.
+
+The fetched container image are stored in a directory in the `WORKDIR` of the
+requesting recipe. When a multi-arch image was specified, only the image
+matching `PACKAGE_ARCH` will be stored. The name of the image directory is
+derived from the container image name, replacing all `/` with `.`.
+
+### Container loader helpers
+
+To create a Debian package which can carry container images and load them into
+local storage of docker or podman, there is a set of helpers available. To use
+them into an own recipe, add
+`require recipes-support/container-loader/docker-loader.inc` when using docker
+and `require recipes-support/container-loader/podman-loader.inc` when using
+podman. The loader will try to transfer the packaged image into the container
+runtime storage on boot, but only if no container image of the same name and
+tag is present already.
+
+Unless `CONTAINER_DELETE_AFTER_LOAD` is set to `1`, the source container images
+remain by default available and may be used again for loading the storage after
+it may have been emptied later on (factory reset).
+
+Source container images may either be fetched as binaries from a registry, see
+above, or built via isar as well.
+
+### Example
+
+This creates debian package with will download, package and then load the
+`debian:bookworm-20240701-slim` container image into the docker container
+storage. The package will depend on `docker.io`, thus ensure that that basic
+runtime services are installed on the target as well. The packaged image will
+be deleted from the target device's rootfs after successful import.
+
+```
+require recipes-support/container-loader/docker-loader.inc
+
+CONTAINER_DELETE_AFTER_LOAD = "1"
+
+SRC_URI += "docker://debian;digest=sha256:f528891ab1aa484bf7233dbcc84f3c806c3e427571d75510a9d74bb5ec535b33;tag=bookworm-20240701-slim"
+```
-- 
2.43.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/2e1537004e7ad48f33cd3d3daf2b8f7c3294d31a.1721038111.git.jan.kiszka%40siemens.com.

Re: [PATCH v2 5/5] doc: Describe how to use the container fetcher and loader

['Niedermayr, BENEDIKT' via isar-users]

On Mon, 2024-07-15 at 12:08 +0200, Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  doc/user_manual.md | 60 ++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 60 insertions(+)
> 
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 776ae52c..733b2b30 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -1519,3 +1519,63 @@ SBUILD_CHROOT_PREINSTALL_EXTRA += "<base packages>"
>  
>  Then, in the dpkg recipe of your package, simply set `SBUILD_FLAVOR = "<your flavor>"`.
>  To install additional packages into the sbuild chroot, add them to
> `SBUILD_CHROOT_PREINSTALL_EXTRA`.
> +
> +## Pre-install container images
> +
> +If an isar-generated image shall provides a container runtime, it may also be

If an isar-generated image shall provide a container runtime...

> +desirable to pre-install container images to avoid having to download them on
> +first boot or because they may not be accessible outside of the build
> +environment. Isar supports this scenario via two services, a container fetcher
> +and a container loader.
> +
> +### Bitbake fetcher for containers
> +
> +The bitbake fetching protocol "docker://" allows to download pre-built images
> +from container registries. The URL consists of the image path, followed by
> +a recommened digests in the form `digest=sha256:<sha256sum>` and an optional
a recommended...

> +tag in the form `tag=<tag>`. A digest is preferred over a tag to identify the
> +an when fetching as it also allows to validate its integrity. If tag is not
... to indentify the ?an? when fetching...

> +specified, `latest` is used as tag name.
> +
> +When specifying a multi-arch image, the fetcher will download the images for
> +all available architectures. If this is not desired, directly specify the
> +digest of the desired architecture manifest instead of that the manifest list.
> +
> +The fetched container image are stored in a directory in the `WORKDIR` of the
> +requesting recipe. When a multi-arch image was specified, only the image
> +matching `PACKAGE_ARCH` will be stored. The name of the image directory is
> +derived from the container image name, replacing all `/` with `.`.
> +
> +### Container loader helpers
> +
> +To create a Debian package which can carry container images and load them into
> +local storage of docker or podman, there is a set of helpers available. To use
> +them into an own recipe, add
> +`require recipes-support/container-loader/docker-loader.inc` when using docker
> +and `require recipes-support/container-loader/podman-loader.inc` when using
> +podman. The loader will try to transfer the packaged image into the container
> +runtime storage on boot, but only if no container image of the same name and
> +tag is present already.
> +
> +Unless `CONTAINER_DELETE_AFTER_LOAD` is set to `1`, the source container images
> +remain by default available and may be used again for loading the storage after
> +it may have been emptied later on (factory reset).
> +
> +Source container images may either be fetched as binaries from a registry, see
> +above, or built via isar as well.
> +
> +### Example
> +
> +This creates debian package with will download, package and then load the
> +`debian:bookworm-20240701-slim` container image into the docker container
> +storage. The package will depend on `docker.io`, thus ensure that that basic
> +runtime services are installed on the target as well. The packaged image will
> +be deleted from the target device's rootfs after successful import.
> +
> +```
> +require recipes-support/container-loader/docker-loader.inc
> +
> +CONTAINER_DELETE_AFTER_LOAD = "1"
> +
> +SRC_URI +=
> "docker://debian;digest=sha256:f528891ab1aa484bf7233dbcc84f3c806c3e427571d75510a9d74bb5ec535b33;ta
> g=bookworm-20240701-slim"
> +```

Benedikt

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/dff2cc09aeee839c613a9d081d63d10553967ccc.camel%40siemens.com.

Re: [PATCH v2 5/5] doc: Describe how to use the container fetcher and loader

['Jan Kiszka' via isar-users]

On 16.07.24 14:56, Niedermayr, Benedikt (T CED OES-DE) wrote:
> On Mon, 2024-07-15 at 12:08 +0200, Jan Kiszka wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>  doc/user_manual.md | 60 ++++++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 60 insertions(+)
>>
>> diff --git a/doc/user_manual.md b/doc/user_manual.md
>> index 776ae52c..733b2b30 100644
>> --- a/doc/user_manual.md
>> +++ b/doc/user_manual.md
>> @@ -1519,3 +1519,63 @@ SBUILD_CHROOT_PREINSTALL_EXTRA += "<base packages>"
>>  
>>  Then, in the dpkg recipe of your package, simply set `SBUILD_FLAVOR = "<your flavor>"`.
>>  To install additional packages into the sbuild chroot, add them to
>> `SBUILD_CHROOT_PREINSTALL_EXTRA`.
>> +
>> +## Pre-install container images
>> +
>> +If an isar-generated image shall provides a container runtime, it may also be
> 
> If an isar-generated image shall provide a container runtime...
> 
>> +desirable to pre-install container images to avoid having to download them on
>> +first boot or because they may not be accessible outside of the build
>> +environment. Isar supports this scenario via two services, a container fetcher
>> +and a container loader.
>> +
>> +### Bitbake fetcher for containers
>> +
>> +The bitbake fetching protocol "docker://" allows to download pre-built images
>> +from container registries. The URL consists of the image path, followed by
>> +a recommened digests in the form `digest=sha256:<sha256sum>` and an optional
> a recommended...
> 
>> +tag in the form `tag=<tag>`. A digest is preferred over a tag to identify the
>> +an when fetching as it also allows to validate its integrity. If tag is not
> ... to indentify the ?an? when fetching...
> 
>> +specified, `latest` is used as tag name.
>> +
>> +When specifying a multi-arch image, the fetcher will download the images for
>> +all available architectures. If this is not desired, directly specify the
>> +digest of the desired architecture manifest instead of that the manifest list.
>> +
>> +The fetched container image are stored in a directory in the `WORKDIR` of the
>> +requesting recipe. When a multi-arch image was specified, only the image
>> +matching `PACKAGE_ARCH` will be stored. The name of the image directory is
>> +derived from the container image name, replacing all `/` with `.`.
>> +
>> +### Container loader helpers
>> +
>> +To create a Debian package which can carry container images and load them into
>> +local storage of docker or podman, there is a set of helpers available. To use
>> +them into an own recipe, add
>> +`require recipes-support/container-loader/docker-loader.inc` when using docker
>> +and `require recipes-support/container-loader/podman-loader.inc` when using
>> +podman. The loader will try to transfer the packaged image into the container
>> +runtime storage on boot, but only if no container image of the same name and
>> +tag is present already.
>> +
>> +Unless `CONTAINER_DELETE_AFTER_LOAD` is set to `1`, the source container images
>> +remain by default available and may be used again for loading the storage after
>> +it may have been emptied later on (factory reset).
>> +
>> +Source container images may either be fetched as binaries from a registry, see
>> +above, or built via isar as well.
>> +
>> +### Example
>> +
>> +This creates debian package with will download, package and then load the
>> +`debian:bookworm-20240701-slim` container image into the docker container
>> +storage. The package will depend on `docker.io`, thus ensure that that basic
>> +runtime services are installed on the target as well. The packaged image will
>> +be deleted from the target device's rootfs after successful import.
>> +
>> +```
>> +require recipes-support/container-loader/docker-loader.inc
>> +
>> +CONTAINER_DELETE_AFTER_LOAD = "1"
>> +
>> +SRC_URI +=
>> "docker://debian;digest=sha256:f528891ab1aa484bf7233dbcc84f3c806c3e427571d75510a9d74bb5ec535b33;ta
>> g=bookworm-20240701-slim"
>> +```
> 
> Benedikt
> 

Thanks for actually reading it :). Update will follow.

Jan

-- 
Siemens AG, Technology
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/59cea3be-00ec-4e8f-a3e8-d16212009e3a%40siemens.com.

RE: [PATCH v2 5/5] doc: Describe how to use the container fetcher and loader

['Schmidt, Adriaan' via isar-users]

'Jan Kiszka' via isar-users <isar-users@googlegroups.com>, Montag, 15. Juli 2024 12:09:
> Subject: [PATCH v2 5/5] doc: Describe how to use the container fetcher and loader
> 
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  doc/user_manual.md | 60
> ++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 60 insertions(+)
> 
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 776ae52c..733b2b30 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -1519,3 +1519,63 @@ SBUILD_CHROOT_PREINSTALL_EXTRA += "<base
> packages>"
> 
>  Then, in the dpkg recipe of your package, simply set `SBUILD_FLAVOR = "<your
> flavor>"`.
>  To install additional packages into the sbuild chroot, add them to
> `SBUILD_CHROOT_PREINSTALL_EXTRA`.
> +
> +## Pre-install container images
> +
> +If an isar-generated image shall provides a container runtime, it may also be
> +desirable to pre-install container images to avoid having to download them on
> +first boot or because they may not be accessible outside of the build
> +environment. Isar supports this scenario via two services, a container fetcher
> +and a container loader.
> +
> +### Bitbake fetcher for containers
> +
> +The bitbake fetching protocol "docker://" allows to download pre-built images
> +from container registries. The URL consists of the image path, followed by
> +a recommened digests in the form `digest=sha256:<sha256sum>` and an optional

recommended *digest

> +tag in the form `tag=<tag>`. A digest is preferred over a tag to identify the
> +an when fetching as it also allows to validate its integrity. If tag is not
> +specified, `latest` is used as tag name.
> +
> +When specifying a multi-arch image, the fetcher will download the images for
> +all available architectures. If this is not desired, directly specify the
> +digest of the desired architecture manifest instead of that the manifest list.
> +
> +The fetched container image are stored in a directory in the `WORKDIR` of the

*images

> +requesting recipe. When a multi-arch image was specified, only the image
> +matching `PACKAGE_ARCH` will be stored. The name of the image directory is
> +derived from the container image name, replacing all `/` with `.`.
> +
> +### Container loader helpers
> +
> +To create a Debian package which can carry container images and load them into
> +local storage of docker or podman, there is a set of helpers available. To use
> +them into an own recipe, add

them *in an own

> +`require recipes-support/container-loader/docker-loader.inc` when using docker
> +and `require recipes-support/container-loader/podman-loader.inc` when using
> +podman. The loader will try to transfer the packaged image into the container
> +runtime storage on boot, but only if no container image of the same name and
> +tag is present already.
> +
> +Unless `CONTAINER_DELETE_AFTER_LOAD` is set to `1`, the source container
> images
> +remain by default available and may be used again for loading the storage after
> +it may have been emptied later on (factory reset).
> +
> +Source container images may either be fetched as binaries from a registry, see
> +above, or built via isar as well.
> +
> +### Example
> +
> +This creates debian package with will download, package and then load the

*which will download

> +`debian:bookworm-20240701-slim` container image into the docker container
> +storage. The package will depend on `docker.io`, thus ensure that that basic

*ensuring?

Adriaan

> +runtime services are installed on the target as well. The packaged image will
> +be deleted from the target device's rootfs after successful import.
> +
> +```
> +require recipes-support/container-loader/docker-loader.inc
> +
> +CONTAINER_DELETE_AFTER_LOAD = "1"
> +
> +SRC_URI +=
> "docker://debian;digest=sha256:f528891ab1aa484bf7233dbcc84f3c806c3e427571d75
> 510a9d74bb5ec535b33;tag=bookworm-20240701-slim"
> +```
> --
> 2.43.0
> 
> --
> You received this message because you are subscribed to the Google Groups "isar-
> users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> isar-users+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/isar-
> users/2e1537004e7ad48f33cd3d3daf2b8f7c3294d31a.1721038111.git.jan.kiszka%40s
> iemens.com.

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/AS4PR10MB5318F184EAC187803BEB67CAEDA22%40AS4PR10MB5318.EURPRD10.PROD.OUTLOOK.COM.