* [PATCH v3 1/5] Introduce fetcher from container registries
2024-07-16 14:18 [PATCH v3 0/5] Introduce container fetcher and pre-loader 'Jan Kiszka' via isar-users
@ 2024-07-16 14:18 ` 'Jan Kiszka' via isar-users
2024-07-17 11:50 ` 'MOESSBAUER, Felix' via isar-users
2024-07-16 14:18 ` [PATCH v3 2/5] container-loader: Introduce helper to load container images into local registry 'Jan Kiszka' via isar-users
` (3 subsequent siblings)
4 siblings, 1 reply; 10+ messages in thread
From: 'Jan Kiszka' via isar-users @ 2024-07-16 14:18 UTC (permalink / raw)
To: isar-users; +Cc: Silvano Cirujano-Cuesta, Benedikt Niedermayr, Felix Moessbauer
From: Jan Kiszka <jan.kiszka@siemens.com>
This bitbake fetcher allows to pull container images from registries,
store them in the download cache and transfer them into the workdir of
recipes requesting the image. The format of the URL is
docker://[<host>/]<image>;digest=sha256:...[;tag=<tag>]
Fetching without digest is supported but will cause a warning, just like
downloading via wget without a checksum. If tag is left out, "latest" is
used.
The fetcher will try to pull all available variants of a multi-arch
image. If this is not needed, you can also directly specify the image
digest of a specific architecture.
Future versions may also introduce full unpacking of the fetched
container layers in workdir if use cases come up.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
meta/classes/dpkg-base.bbclass | 6 +++
meta/lib/container_fetcher.py | 90 ++++++++++++++++++++++++++++++++++
2 files changed, 96 insertions(+)
create mode 100644 meta/lib/container_fetcher.py
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 789d6c74..d90b32a9 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -98,6 +98,12 @@ python() {
if len(d.getVar('SRC_APT').strip()) > 0:
bb.build.addtask('apt_unpack', 'do_patch', '', d)
bb.build.addtask('cleanall_apt', 'do_cleanall', '', d)
+
+ # container docker fetcher
+ import container_fetcher
+ from bb.fetch2 import methods
+
+ methods.append(container_fetcher.Container())
}
do_apt_fetch() {
diff --git a/meta/lib/container_fetcher.py b/meta/lib/container_fetcher.py
new file mode 100644
index 00000000..8513e246
--- /dev/null
+++ b/meta/lib/container_fetcher.py
@@ -0,0 +1,90 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+import os
+import re
+from bb.fetch2 import FetchMethod
+from bb.fetch2 import logger
+from bb.fetch2 import MissingChecksumEvent
+from bb.fetch2 import NoChecksumError
+from bb.fetch2 import runfetchcmd
+from bb.progress import LineFilterProgressHandler
+
+class SkopeoProgressHandler(LineFilterProgressHandler):
+ def __init__(self, d):
+ super(SkopeoProgressHandler, self).__init__(d)
+ self._fire_progress(0)
+
+ def writeline(self, line):
+ match = re.findall(r'^Copying image .*\(([0-9]+/[0-9]+)\)$', line)
+ if match:
+ state = match[0].split('/')
+ progress = (int(state[0]) * 100) / int(state[1])
+ self.update(progress)
+ return True
+
+
+class Container(FetchMethod):
+ def supports(self, ud, d):
+ return ud.type in ['docker']
+
+ def urldata_init(self, ud, d):
+ ud.tag = "latest"
+ if "tag" in ud.parm:
+ ud.tag = ud.parm["tag"]
+
+ ud.digest = None
+ if "digest" in ud.parm:
+ ud.digest = ud.parm["digest"]
+
+ container_name = ud.host + (ud.path if ud.path != "/" else "")
+ ud.container_src = container_name + \
+ ("@" + ud.digest if ud.digest else ":" + ud.tag)
+ ud.localname = container_name.replace('/', '.')
+ ud.localfile = "container-images/" + ud.localname + \
+ "_" + (ud.digest.replace(":", "-") if ud.digest else ud.tag)
+
+ def download(self, ud, d):
+ progresshandler = SkopeoProgressHandler(d)
+ runfetchcmd(f"skopeo copy --preserve-digests --all docker://{ud.container_src} dir:{ud.localfile}",
+ d, log=progresshandler)
+
+ if ud.digest:
+ return
+
+ checksum = bb.utils.sha256_file(ud.localpath + "/manifest.json")
+ checksum_line = f"SRC_URI = \"{ud.url};digest=sha256:{checksum}\""
+
+ strict = d.getVar("BB_STRICT_CHECKSUM") or "0"
+
+ # If strict checking enabled and neither sum defined, raise error
+ if strict == "1":
+ raise NoChecksumError(checksum_line)
+
+ checksum_event = {"sha256sum": checksum}
+ bb.event.fire(MissingChecksumEvent(ud.url, **checksum_event), d)
+
+ if strict == "ignore":
+ return
+
+ # Log missing digest so user can more easily add it
+ logger.warning(
+ f"Missing checksum for '{ud.localpath}', consider using this " \
+ f"SRC_URI in the recipe:\n{checksum_line}")
+
+ def unpack(self, ud, rootdir, d):
+ arch = d.getVar('PACKAGE_ARCH')
+ variant_opt = ""
+ if arch == "armhf":
+ arch = "arm"
+ variant_opt = "--override-variant v7"
+ elif arch == "armel":
+ arch = "arm"
+ variant_opt = "--override-variant v6"
+ runfetchcmd(f"skopeo --override-arch {arch} {variant_opt} " \
+ f"copy dir:{ud.localpath} dir:{rootdir + '/' + ud.localname}", d)
+
+ def clean(self, ud, d):
+ bb.utils.remove(ud.localpath, recurse=True)
--
2.43.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/b0493bec83353990199120a5fcf43676e995bba8.1721139489.git.jan.kiszka%40siemens.com.
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [PATCH v3 1/5] Introduce fetcher from container registries
2024-07-16 14:18 ` [PATCH v3 1/5] Introduce fetcher from container registries 'Jan Kiszka' via isar-users
@ 2024-07-17 11:50 ` 'MOESSBAUER, Felix' via isar-users
2024-07-17 16:02 ` 'Jan Kiszka' via isar-users
0 siblings, 1 reply; 10+ messages in thread
From: 'MOESSBAUER, Felix' via isar-users @ 2024-07-17 11:50 UTC (permalink / raw)
To: isar-users, Kiszka, Jan; +Cc: Niedermayr, BENEDIKT, Cirujano Cuesta, Silvano
On Tue, 2024-07-16 at 16:18 +0200, Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
>
> This bitbake fetcher allows to pull container images from registries,
> store them in the download cache and transfer them into the workdir
> of
> recipes requesting the image. The format of the URL is
>
> docker://[<host>/]<image>;digest=sha256:...[;tag=<tag>]
>
> Fetching without digest is supported but will cause a warning, just
> like
> downloading via wget without a checksum.
This is fine, as long as the upstream artifact is expected to be
stable.
>
> If tag is left out, "latest" is
> used.
The tag should be mandatory and it should be clear that - even without
digest - the artifact needs to be stable. Floating tags create all
kinds of issues (w.r.t. the sstate cache and reproducible builds), so I
vote for just not allowing this.
>
> The fetcher will try to pull all available variants of a multi-arch
> image. If this is not needed, you can also directly specify the image
> digest of a specific architecture.
In most cases this does not make sense. I propose to always limit the
fetching to the current architecture. For that, we either need a
mapping between the debian architecture and the OCI architectures (what
is specified in the application/vnd.oci.image.index.v1+json manifest),
or we simply don't support index manifests at all and force people to
use image manifests (application/vnd.oci.image.manifest.v1+json).
Felix
>
> Future versions may also introduce full unpacking of the fetched
> container layers in workdir if use cases come up.
>
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
> meta/classes/dpkg-base.bbclass | 6 +++
> meta/lib/container_fetcher.py | 90
> ++++++++++++++++++++++++++++++++++
> 2 files changed, 96 insertions(+)
> create mode 100644 meta/lib/container_fetcher.py
>
> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-
> base.bbclass
> index 789d6c74..d90b32a9 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -98,6 +98,12 @@ python() {
> if len(d.getVar('SRC_APT').strip()) > 0:
> bb.build.addtask('apt_unpack', 'do_patch', '', d)
> bb.build.addtask('cleanall_apt', 'do_cleanall', '', d)
> +
> + # container docker fetcher
> + import container_fetcher
> + from bb.fetch2 import methods
> +
> + methods.append(container_fetcher.Container())
> }
>
> do_apt_fetch() {
> diff --git a/meta/lib/container_fetcher.py
> b/meta/lib/container_fetcher.py
> new file mode 100644
> index 00000000..8513e246
> --- /dev/null
> +++ b/meta/lib/container_fetcher.py
> @@ -0,0 +1,90 @@
> +# This software is a part of ISAR.
> +# Copyright (c) Siemens AG, 2024
> +#
> +# SPDX-License-Identifier: MIT
> +
> +import os
> +import re
> +from bb.fetch2 import FetchMethod
> +from bb.fetch2 import logger
> +from bb.fetch2 import MissingChecksumEvent
> +from bb.fetch2 import NoChecksumError
> +from bb.fetch2 import runfetchcmd
> +from bb.progress import LineFilterProgressHandler
> +
> +class SkopeoProgressHandler(LineFilterProgressHandler):
> + def __init__(self, d):
> + super(SkopeoProgressHandler, self).__init__(d)
> + self._fire_progress(0)
> +
> + def writeline(self, line):
> + match = re.findall(r'^Copying image .*\(([0-9]+/[0-9]+)\)$',
> line)
> + if match:
> + state = match[0].split('/')
> + progress = (int(state[0]) * 100) / int(state[1])
> + self.update(progress)
> + return True
> +
> +
> +class Container(FetchMethod):
> + def supports(self, ud, d):
> + return ud.type in ['docker']
> +
> + def urldata_init(self, ud, d):
> + ud.tag = "latest"
> + if "tag" in ud.parm:
> + ud.tag = ud.parm["tag"]
> +
> + ud.digest = None
> + if "digest" in ud.parm:
> + ud.digest = ud.parm["digest"]
> +
> + container_name = ud.host + (ud.path if ud.path != "/" else
> "")
> + ud.container_src = container_name + \
> + ("@" + ud.digest if ud.digest else ":" + ud.tag)
> + ud.localname = container_name.replace('/', '.')
> + ud.localfile = "container-images/" + ud.localname + \
> + "_" + (ud.digest.replace(":", "-") if ud.digest else
> ud.tag)
> +
> + def download(self, ud, d):
> + progresshandler = SkopeoProgressHandler(d)
> + runfetchcmd(f"skopeo copy --preserve-digests --all
> docker://{ud.container_src} dir:{ud.localfile}",
> + d, log=progresshandler)
> +
> + if ud.digest:
> + return
> +
> + checksum = bb.utils.sha256_file(ud.localpath +
> "/manifest.json")
> + checksum_line = f"SRC_URI =
> \"{ud.url};digest=sha256:{checksum}\""
> +
> + strict = d.getVar("BB_STRICT_CHECKSUM") or "0"
> +
> + # If strict checking enabled and neither sum defined, raise
> error
> + if strict == "1":
> + raise NoChecksumError(checksum_line)
> +
> + checksum_event = {"sha256sum": checksum}
> + bb.event.fire(MissingChecksumEvent(ud.url,
> **checksum_event), d)
> +
> + if strict == "ignore":
> + return
> +
> + # Log missing digest so user can more easily add it
> + logger.warning(
> + f"Missing checksum for '{ud.localpath}', consider using
> this " \
> + f"SRC_URI in the recipe:\n{checksum_line}")
> +
> + def unpack(self, ud, rootdir, d):
> + arch = d.getVar('PACKAGE_ARCH')
> + variant_opt = ""
> + if arch == "armhf":
> + arch = "arm"
> + variant_opt = "--override-variant v7"
> + elif arch == "armel":
> + arch = "arm"
> + variant_opt = "--override-variant v6"
> + runfetchcmd(f"skopeo --override-arch {arch} {variant_opt} "
> \
> + f"copy dir:{ud.localpath} dir:{rootdir + '/' +
> ud.localname}", d)
> +
> + def clean(self, ud, d):
> + bb.utils.remove(ud.localpath, recurse=True)
--
Siemens AG, Technology
Linux Expert Center
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/839b07987f40b6edfd06e9d59c4492ee4dfd0edc.camel%40siemens.com.
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [PATCH v3 1/5] Introduce fetcher from container registries
2024-07-17 11:50 ` 'MOESSBAUER, Felix' via isar-users
@ 2024-07-17 16:02 ` 'Jan Kiszka' via isar-users
2024-07-18 7:18 ` 'MOESSBAUER, Felix' via isar-users
0 siblings, 1 reply; 10+ messages in thread
From: 'Jan Kiszka' via isar-users @ 2024-07-17 16:02 UTC (permalink / raw)
To: Moessbauer, Felix (T CED OES-DE), isar-users
Cc: Niedermayr, Benedikt (T CED OES-DE),
Cirujano Cuesta, Silvano (T CED OES-DE)
On 17.07.24 13:50, Moessbauer, Felix (T CED OES-DE) wrote:
> On Tue, 2024-07-16 at 16:18 +0200, Jan Kiszka wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> This bitbake fetcher allows to pull container images from registries,
>> store them in the download cache and transfer them into the workdir
>> of
>> recipes requesting the image. The format of the URL is
>>
>> docker://[<host>/]<image>;digest=sha256:...[;tag=<tag>]
>>
>> Fetching without digest is supported but will cause a warning, just
>> like
>> downloading via wget without a checksum.
>
> This is fine, as long as the upstream artifact is expected to be
> stable.
>
>>
>> If tag is left out, "latest" is
>> used.
>
> The tag should be mandatory and it should be clear that - even without
> digest - the artifact needs to be stable. Floating tags create all
> kinds of issues (w.r.t. the sstate cache and reproducible builds), so I
> vote for just not allowing this.
I cannot follow: Leaving out the tag has nothing to do with getting a
stable image. It may just lead to something being tagged "latest" on the
device that had a different tag (or none) in the registry.
>
>>
>> The fetcher will try to pull all available variants of a multi-arch
>> image. If this is not needed, you can also directly specify the image
>> digest of a specific architecture.
>
> In most cases this does not make sense. I propose to always limit the
> fetching to the current architecture. For that, we either need a
> mapping between the debian architecture and the OCI architectures (what
> is specified in the application/vnd.oci.image.index.v1+json manifest),
> or we simply don't support index manifests at all and force people to
> use image manifests (application/vnd.oci.image.manifest.v1+json).
Don't worry, I already have ideas of redesigning this into per-arch
fetches. Background is using less different formats on the build system
and, where possible, even hard-link between them to save space and time
with larger images.
Jan
--
Siemens AG, Technology
Linux Expert Center
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/e2dff496-ecf9-4f5c-a95d-a9995189b49b%40siemens.com.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v3 1/5] Introduce fetcher from container registries
2024-07-17 16:02 ` 'Jan Kiszka' via isar-users
@ 2024-07-18 7:18 ` 'MOESSBAUER, Felix' via isar-users
0 siblings, 0 replies; 10+ messages in thread
From: 'MOESSBAUER, Felix' via isar-users @ 2024-07-18 7:18 UTC (permalink / raw)
To: isar-users, Kiszka, Jan; +Cc: Niedermayr, BENEDIKT, Cirujano Cuesta, Silvano
On Wed, 2024-07-17 at 18:02 +0200, Jan Kiszka wrote:
> On 17.07.24 13:50, Moessbauer, Felix (T CED OES-DE) wrote:
> > On Tue, 2024-07-16 at 16:18 +0200, Jan Kiszka wrote:
> > > From: Jan Kiszka <jan.kiszka@siemens.com>
> > >
> > > This bitbake fetcher allows to pull container images from
> > > registries,
> > > store them in the download cache and transfer them into the
> > > workdir
> > > of
> > > recipes requesting the image. The format of the URL is
> > >
> > > docker://[<host>/]<image>;digest=sha256:...[;tag=<tag>]
> > >
> > > Fetching without digest is supported but will cause a warning,
> > > just
> > > like
> > > downloading via wget without a checksum.
> >
> > This is fine, as long as the upstream artifact is expected to be
> > stable.
> >
> > >
> > > If tag is left out, "latest" is
> > > used.
> >
> > The tag should be mandatory and it should be clear that - even
> > without
> > digest - the artifact needs to be stable. Floating tags create all
> > kinds of issues (w.r.t. the sstate cache and reproducible builds),
> > so I
> > vote for just not allowing this.
>
> I cannot follow: Leaving out the tag has nothing to do with getting a
> stable image. It may just lead to something being tagged "latest" on
> the
> device that had a different tag (or none) in the registry.
Ok, that means the image is just tagged on the device. Then it should
be fine.
Felix
>
> >
> > >
> > > The fetcher will try to pull all available variants of a multi-
> > > arch
> > > image. If this is not needed, you can also directly specify the
> > > image
> > > digest of a specific architecture.
> >
> > In most cases this does not make sense. I propose to always limit
> > the
> > fetching to the current architecture. For that, we either need a
> > mapping between the debian architecture and the OCI architectures
> > (what
> > is specified in the application/vnd.oci.image.index.v1+json
> > manifest),
> > or we simply don't support index manifests at all and force people
> > to
> > use image manifests (application/vnd.oci.image.manifest.v1+json).
>
> Don't worry, I already have ideas of redesigning this into per-arch
> fetches. Background is using less different formats on the build
> system
> and, where possible, even hard-link between them to save space and
> time
> with larger images.
>
> Jan
>
--
Siemens AG, Technology
Linux Expert Center
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/e082c964a2c3d850566862da82c0c15186fdb55c.camel%40siemens.com.
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH v3 2/5] container-loader: Introduce helper to load container images into local registry
2024-07-16 14:18 [PATCH v3 0/5] Introduce container fetcher and pre-loader 'Jan Kiszka' via isar-users
2024-07-16 14:18 ` [PATCH v3 1/5] Introduce fetcher from container registries 'Jan Kiszka' via isar-users
@ 2024-07-16 14:18 ` 'Jan Kiszka' via isar-users
2024-07-16 20:08 ` 'Niedermayr, BENEDIKT' via isar-users
2024-07-16 14:18 ` [PATCH v3 3/5] meta-isar: Add demo packages for installing prebuilt containers 'Jan Kiszka' via isar-users
` (2 subsequent siblings)
4 siblings, 1 reply; 10+ messages in thread
From: 'Jan Kiszka' via isar-users @ 2024-07-16 14:18 UTC (permalink / raw)
To: isar-users; +Cc: Silvano Cirujano-Cuesta, Benedikt Niedermayr, Felix Moessbauer
From: Jan Kiszka <jan.kiszka@siemens.com>
This allows to write dpkg-raw recipes which packages archived container
images and load them into a local docker or podman registry on boot. The
scenario behind this is to pre-fill local registries in a way that still
permits live updates during runtime.
The loader script only process images which are not yet available under
the same name and tag in the local registry. Also after loading, the
archived images stay on the local file system. This allows to perform
reloading in case the local registry should be emptied (e.g. reset to
factory state). To reduce the space those original images need, they are
compressed, by default with zstd.
Separate include files are available to cater the main container
engines, one for docker and one for podman.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
.../container-loader/container-loader.inc | 101 ++++++++++++++++++
.../container-loader/docker-loader.inc | 10 ++
.../files/container-loader.service.tmpl | 12 +++
.../files/container-loader.sh.tmpl | 18 ++++
.../container-loader/podman-loader.inc | 10 ++
5 files changed, 151 insertions(+)
create mode 100644 meta/recipes-support/container-loader/container-loader.inc
create mode 100644 meta/recipes-support/container-loader/docker-loader.inc
create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl
create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl
create mode 100644 meta/recipes-support/container-loader/podman-loader.inc
diff --git a/meta/recipes-support/container-loader/container-loader.inc b/meta/recipes-support/container-loader/container-loader.inc
new file mode 100644
index 00000000..e97e829b
--- /dev/null
+++ b/meta/recipes-support/container-loader/container-loader.inc
@@ -0,0 +1,101 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+FILESPATH:append := ":${FILE_DIRNAME}/files"
+
+inherit dpkg-raw
+
+SRC_URI += " \
+ file://container-loader.service.tmpl \
+ file://container-loader.sh.tmpl"
+
+CONTAINER_COMPRESSION ?= "zst"
+CONTAINER_DELETE_AFTER_LOAD ?= "0"
+
+DEBIAN_DEPENDS += " \
+ ${CONTAINER_ENGINE_PACKAGES} \
+ ${@', gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
+ ', zstd' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \
+ ', xz-utils' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
+ ''}"
+
+CONTAINER_COMPRESSOR_CMD = "${@ \
+ 'gzip -f -9 -n --rsyncable' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
+ 'xz -f ${XZ_DEFAULTS}' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
+ 'zstd -f --rm ${ZSTD_DEFAULTS}' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \
+ ''}"
+
+CONTAINER_DECOMPRESSOR_CMD = "${@ \
+ 'gzip -c -d -n' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
+ 'xz -c -d -T0' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
+ 'pzstd -c -d' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \
+ ''}"
+
+TEMPLATE_FILES += " \
+ container-loader.service.tmpl \
+ container-loader.sh.tmpl"
+TEMPLATE_VARS += " \
+ CONTAINER_ENGINE \
+ CONTAINER_DECOMPRESSOR_CMD \
+ CONTAINER_DELETE_AFTER_LOAD"
+
+do_install() {
+ install -m 755 ${WORKDIR}/container-loader.sh ${D}/usr/share/${BPN}
+}
+do_install[cleandirs] += " \
+ ${D}/usr/share/${BPN} \
+ ${D}/usr/share/${BPN}/images"
+
+python do_install_fetched_containers() {
+ import os
+
+ workdir = d.getVar('WORKDIR')
+ D = d.getVar('D')
+ BPN = d.getVar('BPN')
+
+ image_list = open(D + "/usr/share/" + BPN + "/image.list", "w")
+
+ src_uri = d.getVar('SRC_URI').split()
+ for uri in src_uri:
+ scheme, host, path, _, _, parm = bb.fetch.decodeurl(uri)
+ if scheme != "docker":
+ continue
+
+ image_name = host + (path if path != "/" else "")
+ unpacked_image = workdir + "/" + image_name.replace('/', '.')
+ dest_dir = D + "/usr/share/" + BPN + "/images"
+ tar_image = dest_dir + "/" + image_name.replace('/', '.') + ".tar"
+ docker_ref = ":" + parm["tag"] if "tag" in parm else "latest"
+
+ bb.utils.remove(tar_image)
+ cmd = f"skopeo copy dir:{unpacked_image} " \
+ f"docker-archive:{tar_image}:{image_name}{docker_ref}"
+ bb.note(f"running: {cmd}")
+ bb.process.run(cmd)
+
+ cmd = f"{d.getVar('CONTAINER_COMPRESSOR_CMD')} {tar_image}"
+ bb.note(f"running: {cmd}")
+ bb.process.run(cmd)
+
+ line = f"{os.path.basename(tar_image)}.{d.getVar('CONTAINER_COMPRESSION')} " + \
+ image_name + docker_ref
+ bb.note(f"adding '{line}' to image.list")
+ image_list.write(line + "\n")
+
+ image_list.close()
+}
+
+addtask install_fetched_containers after do_install before do_prepare_build
+
+do_prepare_build:append() {
+ install -v -m 644 ${WORKDIR}/container-loader.service ${S}/debian/${BPN}.service
+
+ # Do not compress the package, most of its payload is already, and trying
+ # nevertheless will only cost time without any gain.
+ cat <<EOF >> ${S}/debian/rules
+override_dh_builddeb:
+ dh_builddeb -- -Znone
+EOF
+}
diff --git a/meta/recipes-support/container-loader/docker-loader.inc b/meta/recipes-support/container-loader/docker-loader.inc
new file mode 100644
index 00000000..b864c854
--- /dev/null
+++ b/meta/recipes-support/container-loader/docker-loader.inc
@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+require container-loader.inc
+
+CONTAINER_ENGINE = "docker"
+
+CONTAINER_ENGINE_PACKAGES ?= "docker.io, apparmor"
diff --git a/meta/recipes-support/container-loader/files/container-loader.service.tmpl b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
new file mode 100644
index 00000000..1638eaf2
--- /dev/null
+++ b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
@@ -0,0 +1,12 @@
+[Unit]
+Description=Load archived container images on boot
+After=${CONTAINER_ENGINE}.service
+Requires=${CONTAINER_ENGINE}.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/share/${BPN}/container-loader.sh
+RemainAfterExit=true
+
+[Install]
+WantedBy=multi-user.target
diff --git a/meta/recipes-support/container-loader/files/container-loader.sh.tmpl b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl
new file mode 100755
index 00000000..b6abec92
--- /dev/null
+++ b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl
@@ -0,0 +1,18 @@
+#!/bin/sh
+#
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+set -eu
+
+while read -r image ref; do
+ if [ -e /usr/share/${BPN}/images/"$image" ] && \
+ [ -z "$(${CONTAINER_ENGINE} images -q "$ref")" ]; then
+ ${CONTAINER_DECOMPRESSOR_CMD} /usr/share/${BPN}/images/"$image" | \
+ ${CONTAINER_ENGINE} load
+ if [ "${CONTAINER_DELETE_AFTER_LOAD}" = "1" ]; then
+ rm -f /usr/share/${BPN}/images/"$image"
+ fi
+ fi
+done < /usr/share/${BPN}/image.list
diff --git a/meta/recipes-support/container-loader/podman-loader.inc b/meta/recipes-support/container-loader/podman-loader.inc
new file mode 100644
index 00000000..d2c9a12d
--- /dev/null
+++ b/meta/recipes-support/container-loader/podman-loader.inc
@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+require container-loader.inc
+
+CONTAINER_ENGINE = "podman"
+
+CONTAINER_ENGINE_PACKAGES ?= "podman"
--
2.43.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/3b7b8dbdde7fa3a4184daa3f8d567e72c8b50d2d.1721139489.git.jan.kiszka%40siemens.com.
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [PATCH v3 2/5] container-loader: Introduce helper to load container images into local registry
2024-07-16 14:18 ` [PATCH v3 2/5] container-loader: Introduce helper to load container images into local registry 'Jan Kiszka' via isar-users
@ 2024-07-16 20:08 ` 'Niedermayr, BENEDIKT' via isar-users
0 siblings, 0 replies; 10+ messages in thread
From: 'Niedermayr, BENEDIKT' via isar-users @ 2024-07-16 20:08 UTC (permalink / raw)
To: isar-users, Kiszka, Jan; +Cc: Cirujano Cuesta, Silvano, MOESSBAUER, Felix
On Tue, 2024-07-16 at 16:18 +0200, Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
>
> This allows to write dpkg-raw recipes which packages archived container
> images and load them into a local docker or podman registry on boot. The
> scenario behind this is to pre-fill local registries in a way that still
> permits live updates during runtime.
>
> The loader script only process images which are not yet available under
> the same name and tag in the local registry. Also after loading, the
> archived images stay on the local file system. This allows to perform
> reloading in case the local registry should be emptied (e.g. reset to
> factory state). To reduce the space those original images need, they are
> compressed, by default with zstd.
>
> Separate include files are available to cater the main container
> engines, one for docker and one for podman.
>
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
> .../container-loader/container-loader.inc | 101 ++++++++++++++++++
> .../container-loader/docker-loader.inc | 10 ++
> .../files/container-loader.service.tmpl | 12 +++
> .../files/container-loader.sh.tmpl | 18 ++++
> .../container-loader/podman-loader.inc | 10 ++
> 5 files changed, 151 insertions(+)
> create mode 100644 meta/recipes-support/container-loader/container-loader.inc
> create mode 100644 meta/recipes-support/container-loader/docker-loader.inc
> create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl
> create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl
> create mode 100644 meta/recipes-support/container-loader/podman-loader.inc
>
> diff --git a/meta/recipes-support/container-loader/container-loader.inc b/meta/recipes-
> support/container-loader/container-loader.inc
> new file mode 100644
> index 00000000..e97e829b
> --- /dev/null
> +++ b/meta/recipes-support/container-loader/container-loader.inc
> @@ -0,0 +1,101 @@
> +# This software is a part of ISAR.
> +# Copyright (c) Siemens AG, 2024
> +#
> +# SPDX-License-Identifier: MIT
> +
> +FILESPATH:append := ":${FILE_DIRNAME}/files"
> +
> +inherit dpkg-raw
> +
> +SRC_URI += " \
> + file://container-loader.service.tmpl \
> + file://container-loader.sh.tmpl"
> +
> +CONTAINER_COMPRESSION ?= "zst"
> +CONTAINER_DELETE_AFTER_LOAD ?= "0"
> +
> +DEBIAN_DEPENDS += " \
> + ${CONTAINER_ENGINE_PACKAGES} \
> + ${@', gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
> + ', zstd' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \
> + ', xz-utils' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
> + ''}"
> +
> +CONTAINER_COMPRESSOR_CMD = "${@ \
> + 'gzip -f -9 -n --rsyncable' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
> + 'xz -f ${XZ_DEFAULTS}' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
> + 'zstd -f --rm ${ZSTD_DEFAULTS}' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \
> + ''}"
> +
> +CONTAINER_DECOMPRESSOR_CMD = "${@ \
> + 'gzip -c -d -n' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
> + 'xz -c -d -T0' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
> + 'pzstd -c -d' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \
> + ''}"
> +
> +TEMPLATE_FILES += " \
> + container-loader.service.tmpl \
> + container-loader.sh.tmpl"
> +TEMPLATE_VARS += " \
> + CONTAINER_ENGINE \
> + CONTAINER_DECOMPRESSOR_CMD \
> + CONTAINER_DELETE_AFTER_LOAD"
> +
> +do_install() {
> + install -m 755 ${WORKDIR}/container-loader.sh ${D}/usr/share/${BPN}
> +}
> +do_install[cleandirs] += " \
> + ${D}/usr/share/${BPN} \
> + ${D}/usr/share/${BPN}/images"
> +
> +python do_install_fetched_containers() {
> + import os
> +
> + workdir = d.getVar('WORKDIR')
> + D = d.getVar('D')
> + BPN = d.getVar('BPN')
> +
> + image_list = open(D + "/usr/share/" + BPN + "/image.list", "w")
> +
> + src_uri = d.getVar('SRC_URI').split()
> + for uri in src_uri:
> + scheme, host, path, _, _, parm = bb.fetch.decodeurl(uri)
> + if scheme != "docker":
> + continue
> +
> + image_name = host + (path if path != "/" else "")
> + unpacked_image = workdir + "/" + image_name.replace('/', '.')
> + dest_dir = D + "/usr/share/" + BPN + "/images"
> + tar_image = dest_dir + "/" + image_name.replace('/', '.') + ".tar"
> + docker_ref = ":" + parm["tag"] if "tag" in parm else "latest"
> +
> + bb.utils.remove(tar_image)
> + cmd = f"skopeo copy dir:{unpacked_image} " \
> + f"docker-archive:{tar_image}:{image_name}{docker_ref}"
> + bb.note(f"running: {cmd}")
> + bb.process.run(cmd)
> +
> + cmd = f"{d.getVar('CONTAINER_COMPRESSOR_CMD')} {tar_image}"
> + bb.note(f"running: {cmd}")
> + bb.process.run(cmd)
> +
> + line = f"{os.path.basename(tar_image)}.{d.getVar('CONTAINER_COMPRESSION')} " + \
> + image_name + docker_ref
> + bb.note(f"adding '{line}' to image.list")
> + image_list.write(line + "\n")
> +
> + image_list.close()
> +}
> +
> +addtask install_fetched_containers after do_install before do_prepare_build
Add do_install_fetched_containers[vardeps] += "CONTAINER_COMPRESSOR_CMD" here.
By the way, for this to work the implementation in bitbake.conf needs to be fixed as well.
The vardepsexclude for ZSTD_DEFAULTS should not contain "ZSTD_LEVEL".
I could sent out a patch for this.
Benedikt
> +
> +do_prepare_build:append() {
> + install -v -m 644 ${WORKDIR}/container-loader.service ${S}/debian/${BPN}.service
> +
> + # Do not compress the package, most of its payload is already, and trying
> + # nevertheless will only cost time without any gain.
> + cat <<EOF >> ${S}/debian/rules
> +override_dh_builddeb:
> + dh_builddeb -- -Znone
> +EOF
> +}
> diff --git a/meta/recipes-support/container-loader/docker-loader.inc b/meta/recipes-
> support/container-loader/docker-loader.inc
> new file mode 100644
> index 00000000..b864c854
> --- /dev/null
> +++ b/meta/recipes-support/container-loader/docker-loader.inc
> @@ -0,0 +1,10 @@
> +# This software is a part of ISAR.
> +# Copyright (c) Siemens AG, 2024
> +#
> +# SPDX-License-Identifier: MIT
> +
> +require container-loader.inc
> +
> +CONTAINER_ENGINE = "docker"
> +
> +CONTAINER_ENGINE_PACKAGES ?= "docker.io, apparmor"
> diff --git a/meta/recipes-support/container-loader/files/container-loader.service.tmpl
> b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
> new file mode 100644
> index 00000000..1638eaf2
> --- /dev/null
> +++ b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
> @@ -0,0 +1,12 @@
> +[Unit]
> +Description=Load archived container images on boot
> +After=${CONTAINER_ENGINE}.service
> +Requires=${CONTAINER_ENGINE}.service
> +
> +[Service]
> +Type=oneshot
> +ExecStart=/usr/share/${BPN}/container-loader.sh
> +RemainAfterExit=true
> +
> +[Install]
> +WantedBy=multi-user.target
> diff --git a/meta/recipes-support/container-loader/files/container-loader.sh.tmpl b/meta/recipes-
> support/container-loader/files/container-loader.sh.tmpl
> new file mode 100755
> index 00000000..b6abec92
> --- /dev/null
> +++ b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl
> @@ -0,0 +1,18 @@
> +#!/bin/sh
> +#
> +# Copyright (c) Siemens AG, 2024
> +#
> +# SPDX-License-Identifier: MIT
> +
> +set -eu
> +
> +while read -r image ref; do
> + if [ -e /usr/share/${BPN}/images/"$image" ] && \
> + [ -z "$(${CONTAINER_ENGINE} images -q "$ref")" ]; then
> + ${CONTAINER_DECOMPRESSOR_CMD} /usr/share/${BPN}/images/"$image" | \
> + ${CONTAINER_ENGINE} load
> + if [ "${CONTAINER_DELETE_AFTER_LOAD}" = "1" ]; then
> + rm -f /usr/share/${BPN}/images/"$image"
> + fi
> + fi
> +done < /usr/share/${BPN}/image.list
> diff --git a/meta/recipes-support/container-loader/podman-loader.inc b/meta/recipes-
> support/container-loader/podman-loader.inc
> new file mode 100644
> index 00000000..d2c9a12d
> --- /dev/null
> +++ b/meta/recipes-support/container-loader/podman-loader.inc
> @@ -0,0 +1,10 @@
> +# This software is a part of ISAR.
> +# Copyright (c) Siemens AG, 2024
> +#
> +# SPDX-License-Identifier: MIT
> +
> +require container-loader.inc
> +
> +CONTAINER_ENGINE = "podman"
> +
> +CONTAINER_ENGINE_PACKAGES ?= "podman"
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/f73b26066edb422e5add7f67098cc53bc12fb2b4.camel%40siemens.com.
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH v3 3/5] meta-isar: Add demo packages for installing prebuilt containers
2024-07-16 14:18 [PATCH v3 0/5] Introduce container fetcher and pre-loader 'Jan Kiszka' via isar-users
2024-07-16 14:18 ` [PATCH v3 1/5] Introduce fetcher from container registries 'Jan Kiszka' via isar-users
2024-07-16 14:18 ` [PATCH v3 2/5] container-loader: Introduce helper to load container images into local registry 'Jan Kiszka' via isar-users
@ 2024-07-16 14:18 ` 'Jan Kiszka' via isar-users
2024-07-16 14:18 ` [PATCH v3 4/5] ci: Add test cases for container fetching and loading 'Jan Kiszka' via isar-users
2024-07-16 14:18 ` [PATCH v3 5/5] doc: Describe how to use the container fetcher and loader 'Jan Kiszka' via isar-users
4 siblings, 0 replies; 10+ messages in thread
From: 'Jan Kiszka' via isar-users @ 2024-07-16 14:18 UTC (permalink / raw)
To: isar-users; +Cc: Silvano Cirujano-Cuesta, Benedikt Niedermayr, Felix Moessbauer
From: Jan Kiszka <jan.kiszka@siemens.com>
One recipe for docker, one for podman. Both pull from a registry that,
in contrast to infamous dockerhub, should not throttle CI jobs running
these frequently for testing purposes.
The podman variant of the recipe is intentionally leaving out the digest
to trigger the related warning of the container fetcher.
These demos also come with kas integration.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
kas/package/Kconfig | 19 +++++++++++++++++++
kas/package/pkg_prebuilt-docker-img.yaml | 9 +++++++++
kas/package/pkg_prebuilt-podman-img.yaml | 9 +++++++++
.../prebuilt-docker-img_0.1.bb | 12 ++++++++++++
.../prebuilt-podman-img_0.1.bb | 10 ++++++++++
5 files changed, 59 insertions(+)
create mode 100644 kas/package/pkg_prebuilt-docker-img.yaml
create mode 100644 kas/package/pkg_prebuilt-podman-img.yaml
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb
diff --git a/kas/package/Kconfig b/kas/package/Kconfig
index 35ba7cf1..395c3a08 100644
--- a/kas/package/Kconfig
+++ b/kas/package/Kconfig
@@ -146,6 +146,25 @@ config KAS_INCLUDE_PACKAGE_ISAR_CI_SSH_SETUP
default "kas/package/pkg_sshd-regen-keys.yaml"
depends on PACKAGE_ISAR_CI_SSH_SETUP
+
+config PACKAGE_PREBUILT_DOCKER_IMG
+ bool "prebuilt-docker-img"
+ default y
+
+config KAS_INCLUDE_PACKAGE_PREBUILT_DOCKER_IMG
+ string
+ default "kas/package/pkg_prebuilt-docker-img.yaml"
+ depends on PACKAGE_PREBUILT_DOCKER_IMG
+
+config PACKAGE_PREBUILT_PODMAN_IMG
+ bool "prebuilt-podman-img"
+ default y
+
+config KAS_INCLUDE_PACKAGE_PREBUILT_PODMAN_IMG
+ string
+ default "kas/package/pkg_prebuilt-podman-img.yaml"
+ depends on PACKAGE_PREBUILT_PODMAN_IMG
+
endmenu
config KAS_IMAGE_PREINSTALL
diff --git a/kas/package/pkg_prebuilt-docker-img.yaml b/kas/package/pkg_prebuilt-docker-img.yaml
new file mode 100644
index 00000000..df96a484
--- /dev/null
+++ b/kas/package/pkg_prebuilt-docker-img.yaml
@@ -0,0 +1,9 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+
+header:
+ version: 14
+
+local_conf_header:
+ package-prebuilt-docker-img: |
+ IMAGE_INSTALL:append = " prebuilt-docker-img"
diff --git a/kas/package/pkg_prebuilt-podman-img.yaml b/kas/package/pkg_prebuilt-podman-img.yaml
new file mode 100644
index 00000000..d0b8da1c
--- /dev/null
+++ b/kas/package/pkg_prebuilt-podman-img.yaml
@@ -0,0 +1,9 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+
+header:
+ version: 14
+
+local_conf_header:
+ package-prebuilt-podman-img: |
+ IMAGE_INSTALL:append = " prebuilt-podman-img"
diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
new file mode 100644
index 00000000..0dfc9b8f
--- /dev/null
+++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
@@ -0,0 +1,12 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+require recipes-support/container-loader/docker-loader.inc
+
+CONTAINER_DELETE_AFTER_LOAD = "1"
+
+SRC_URI += "\
+ docker://quay.io/libpod/alpine;digest=sha256:fa93b01658e3a5a1686dc3ae55f170d8de487006fb53a28efcd12ab0710a2e5f;tag=3.10.2 \
+ "
diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb
new file mode 100644
index 00000000..e671a494
--- /dev/null
+++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb
@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+require recipes-support/container-loader/podman-loader.inc
+
+SRC_URI += "\
+ docker://quay.io/libpod/alpine;tag=latest \
+ "
--
2.43.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/cf30840b8cdd35f2e959c40710a568381403b689.1721139489.git.jan.kiszka%40siemens.com.
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH v3 4/5] ci: Add test cases for container fetching and loading
2024-07-16 14:18 [PATCH v3 0/5] Introduce container fetcher and pre-loader 'Jan Kiszka' via isar-users
` (2 preceding siblings ...)
2024-07-16 14:18 ` [PATCH v3 3/5] meta-isar: Add demo packages for installing prebuilt containers 'Jan Kiszka' via isar-users
@ 2024-07-16 14:18 ` 'Jan Kiszka' via isar-users
2024-07-16 14:18 ` [PATCH v3 5/5] doc: Describe how to use the container fetcher and loader 'Jan Kiszka' via isar-users
4 siblings, 0 replies; 10+ messages in thread
From: 'Jan Kiszka' via isar-users @ 2024-07-16 14:18 UTC (permalink / raw)
To: isar-users; +Cc: Silvano Cirujano-Cuesta, Benedikt Niedermayr, Felix Moessbauer
From: Jan Kiszka <jan.kiszka@siemens.com>
This plugs the two example recipes for loading container images into
VM-based testing. The test consists of running 'true' in the installed
alpine images.
Rather than enabling the ci user to do password-less sudo, this uses su
with the piped-in password. Another trick needed is to poll for the
images because loading is performed asynchronously.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
.../recipes-core/images/isar-image-ci.bb | 2 ++
testsuite/citest.py | 21 +++++++++++++++++++
2 files changed, 23 insertions(+)
diff --git a/meta-test/recipes-core/images/isar-image-ci.bb b/meta-test/recipes-core/images/isar-image-ci.bb
index e5d51e6e..9133da74 100644
--- a/meta-test/recipes-core/images/isar-image-ci.bb
+++ b/meta-test/recipes-core/images/isar-image-ci.bb
@@ -16,6 +16,7 @@ IMAGE_INSTALL += "sshd-regen-keys"
# qemuamd64-bookworm
WKS_FILE:qemuamd64:debian-bookworm ?= "multipart-efi.wks"
+IMAGE_INSTALL:append:qemuamd64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img"
# qemuamd64-bullseye
IMAGE_FSTYPES:append:qemuamd64:debian-bullseye ?= " cpio.gz tar.gz"
@@ -51,3 +52,4 @@ IMAGER_INSTALL:append:qemuarm:debian-bookworm ?= " ${SYSTEMD_BOOTLOADER_INSTALL}
# qemuarm64-bookworm
IMAGE_FSTYPES:append:qemuarm64:debian-bookworm ?= " wic.xz"
IMAGER_INSTALL:append:qemuarm64:debian-bookworm ?= " ${GRUB_BOOTLOADER_INSTALL}"
+IMAGE_INSTALL:append:qemuarm64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img"
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 8dd907d0..539c9440 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -522,3 +522,24 @@ class VmBootTestFull(CIBaseTest):
self.init()
self.vm_start('mipsel','bookworm', image='isar-image-ci',
script='test_kernel_module.sh example_module')
+
+
+ def test_amd64_bookworm_prebuilt_containers(self):
+ self.init()
+ self.vm_start('amd64', 'bookworm', image='isar-image-ci',
+ cmd='echo root | su -c \'' \
+ 'PATH=\$PATH:/usr/sbin;' \
+ 'for n in \$(seq 30); do docker images | grep -q alpine && break; sleep 10; done;' \
+ 'docker run --rm quay.io/libpod/alpine:3.10.2 true && ' \
+ 'for n in \$(seq 30); do podman images | grep -q alpine && break; sleep 10; done;' \
+ 'podman run --rm quay.io/libpod/alpine:latest true\'')
+
+ def test_arm64_bookworm_prebuilt_containers(self):
+ self.init()
+ self.vm_start('arm64', 'bookworm', image='isar-image-ci',
+ cmd='echo root | su -c \'' \
+ 'PATH=\$PATH:/usr/sbin;' \
+ 'for n in \$(seq 30); do docker images | grep -q alpine && break; sleep 10; done;' \
+ 'docker run --rm quay.io/libpod/alpine:3.10.2 true && ' \
+ 'for n in \$(seq 30); do podman images | grep -q alpine && break; sleep 10; done;' \
+ 'podman run --rm quay.io/libpod/alpine:latest true\'')
--
2.43.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/2bf448610b18ca3ecc438f2a8342db7da0dacf02.1721139489.git.jan.kiszka%40siemens.com.
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH v3 5/5] doc: Describe how to use the container fetcher and loader
2024-07-16 14:18 [PATCH v3 0/5] Introduce container fetcher and pre-loader 'Jan Kiszka' via isar-users
` (3 preceding siblings ...)
2024-07-16 14:18 ` [PATCH v3 4/5] ci: Add test cases for container fetching and loading 'Jan Kiszka' via isar-users
@ 2024-07-16 14:18 ` 'Jan Kiszka' via isar-users
4 siblings, 0 replies; 10+ messages in thread
From: 'Jan Kiszka' via isar-users @ 2024-07-16 14:18 UTC (permalink / raw)
To: isar-users; +Cc: Silvano Cirujano-Cuesta, Benedikt Niedermayr, Felix Moessbauer
From: Jan Kiszka <jan.kiszka@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
doc/user_manual.md | 60 ++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 60 insertions(+)
diff --git a/doc/user_manual.md b/doc/user_manual.md
index 776ae52c..e97a2cd5 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -1519,3 +1519,63 @@ SBUILD_CHROOT_PREINSTALL_EXTRA += "<base packages>"
Then, in the dpkg recipe of your package, simply set `SBUILD_FLAVOR = "<your flavor>"`.
To install additional packages into the sbuild chroot, add them to `SBUILD_CHROOT_PREINSTALL_EXTRA`.
+
+## Pre-install container images
+
+If an isar-generated image shall provide a container runtime, it may also be
+desirable to pre-install container images to avoid having to download them on
+first boot or because they may not be accessible outside of the build
+environment. Isar supports this scenario via two services, a container fetcher
+and a container loader.
+
+### Bitbake fetcher for containers
+
+The bitbake fetching protocol "docker://" allows to download pre-built images
+from container registries. The URL consists of the image path, followed by
+a recommended digest in the form `digest=sha256:<sha256sum>` and an optional
+tag in the form `tag=<tag>`. A digest is preferred over a tag to identify an
+image when fetching because it also allows to validate its integrity. If a tag
+is not specified, `latest` is used as tag name.
+
+When specifying a multi-arch image, the fetcher will download the images for
+all available architectures. If this is not desired, directly specify the
+digest of the desired architecture manifest instead of that the manifest list.
+
+The fetched container images are stored in a directory in the `WORKDIR` of the
+requesting recipe. When a multi-arch image was specified, only the image
+matching `PACKAGE_ARCH` will be stored. The name of the image directory is
+derived from the container image name, replacing all `/` with `.`.
+
+### Container loader helpers
+
+To create a Debian package which can carry container images and load them into
+local storage of docker or podman, there is a set of helpers available. To use
+them in an own recipe, add
+`require recipes-support/container-loader/docker-loader.inc` when using docker
+and `require recipes-support/container-loader/podman-loader.inc` when using
+podman. The loader will try to transfer the packaged image into the container
+runtime storage on boot, but only if no container image of the same name and
+tag is present already.
+
+Unless `CONTAINER_DELETE_AFTER_LOAD` is set to `1`, the source container images
+remain by default available and may be used again for loading the storage after
+it may have been emptied later on (factory reset).
+
+Source container images may either be fetched as binaries from a registry, see
+above, or built via isar as well.
+
+### Example
+
+This creates a debian package which will download, package and then load the
+`debian:bookworm-20240701-slim` container image into the docker container
+storage. The package will depend on `docker.io`, insuring that that basic
+runtime services are installed on the target as well. The packaged image will
+be deleted from the target device's rootfs after successful import.
+
+```
+require recipes-support/container-loader/docker-loader.inc
+
+CONTAINER_DELETE_AFTER_LOAD = "1"
+
+SRC_URI += "docker://debian;digest=sha256:f528891ab1aa484bf7233dbcc84f3c806c3e427571d75510a9d74bb5ec535b33;tag=bookworm-20240701-slim"
+```
--
2.43.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/80f1c915285e61b70623f0eb2ce5f930abaa6196.1721139489.git.jan.kiszka%40siemens.com.
^ permalink raw reply [flat|nested] 10+ messages in thread