* [PATCH v4 1/5] Introduce fetcher from container registries
2024-07-19 16:38 [PATCH v4 0/5] Introduce container fetcher and pre-loader 'Jan Kiszka' via isar-users
@ 2024-07-19 16:38 ` 'Jan Kiszka' via isar-users
2024-07-25 10:48 ` 'Niedermayr, BENEDIKT' via isar-users
2024-07-19 16:38 ` [PATCH v4 2/5] container-loader: Introduce helper to load container images into local registry 'Jan Kiszka' via isar-users
` (4 subsequent siblings)
5 siblings, 1 reply; 12+ messages in thread
From: 'Jan Kiszka' via isar-users @ 2024-07-19 16:38 UTC (permalink / raw)
To: isar-users; +Cc: Silvano Cirujano-Cuesta, Benedikt Niedermayr, Felix Moessbauer
From: Jan Kiszka <jan.kiszka@siemens.com>
This bitbake fetcher allows to pull container images from registries,
store them in the download cache and transfer them into the workdir of
recipes requesting the image. The format of the URL is
docker://[<host>/]<image>;digest=sha256:...[;tag=<tag>]
Fetching without digest is supported but will cause a warning, just like
downloading via wget without a checksum. If tag is left out, "latest" is
used.
In case a multi-arch image is specified, the fetcher will only pull for
the package architecture of the requesting recipe. The image is stored
compressed in docker-archive format and, wherever possible, hard-linked
from DL_DIR to WORKDIR. Future versions may also introduce full
unpacking of the fetched container layers in workdir if use cases come up.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
meta/classes/dpkg-base.bbclass | 6 +++
meta/lib/container_fetcher.py | 86 ++++++++++++++++++++++++++++++++++
2 files changed, 92 insertions(+)
create mode 100644 meta/lib/container_fetcher.py
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 789d6c74..d90b32a9 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -98,6 +98,12 @@ python() {
if len(d.getVar('SRC_APT').strip()) > 0:
bb.build.addtask('apt_unpack', 'do_patch', '', d)
bb.build.addtask('cleanall_apt', 'do_cleanall', '', d)
+
+ # container docker fetcher
+ import container_fetcher
+ from bb.fetch2 import methods
+
+ methods.append(container_fetcher.Container())
}
do_apt_fetch() {
diff --git a/meta/lib/container_fetcher.py b/meta/lib/container_fetcher.py
new file mode 100644
index 00000000..0d659154
--- /dev/null
+++ b/meta/lib/container_fetcher.py
@@ -0,0 +1,86 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+import oe.path
+import os
+import tempfile
+from bb.fetch2 import FetchMethod
+from bb.fetch2 import logger
+from bb.fetch2 import MissingChecksumEvent
+from bb.fetch2 import NoChecksumError
+from bb.fetch2 import runfetchcmd
+
+class Container(FetchMethod):
+ def supports(self, ud, d):
+ return ud.type in ['docker']
+
+ def urldata_init(self, ud, d):
+ ud.tag = "latest"
+ if "tag" in ud.parm:
+ ud.tag = ud.parm["tag"]
+
+ ud.digest = None
+ if "digest" in ud.parm:
+ ud.digest = ud.parm["digest"]
+
+ ud.arch = d.getVar('PACKAGE_ARCH')
+ ud.variant = None
+ if ud.arch == "armhf":
+ ud.arch = "arm"
+ ud.variant = "v7"
+ elif ud.arch == "armel":
+ ud.arch = "arm"
+ ud.variant = "v6"
+
+ ud.container_name = ud.host + (ud.path if ud.path != "/" else "")
+ ud.container_src = ud.container_name + \
+ ("@" + ud.digest if ud.digest else ":" + ud.tag)
+ ud.localname = ud.container_name.replace('/', '.')
+ ud.localfile = "container-images/" + ud.arch + "/" + \
+ (ud.variant + "/" if ud.variant else "") + ud.localname + \
+ "_" + (ud.digest.replace(":", "-") if ud.digest else ud.tag) + \
+ ".zst"
+
+ def download(self, ud, d):
+ tarball = ud.localfile[:-len('.zst')]
+ with tempfile.TemporaryDirectory(dir=d.getVar('DL_DIR')) as tmpdir:
+ # Take a two steps for downloading into a docker archive because
+ # not all source may have the required Docker schema 2 manifest.
+ runfetchcmd("skopeo copy --preserve-digests " + \
+ f"--override-arch {ud.arch} " + \
+ (f"--override-variant {ud.variant} " if ud.variant else "") + \
+ f"docker://{ud.container_src} dir:{tmpdir}", d)
+ runfetchcmd(f"skopeo copy dir:{tmpdir} " + \
+ f"docker-archive:{tarball}:{ud.container_name}:{ud.tag}", d)
+ zstd_defaults = d.getVar('ZSTD_DEFAULTS')
+ runfetchcmd(f"zstd -f --rm {zstd_defaults} {tarball}", d)
+
+ if ud.digest:
+ return
+
+ checksum = bb.utils.sha256_file(ud.localpath + "/manifest.json")
+ checksum_line = f"SRC_URI = \"{ud.url};digest=sha256:{checksum}\""
+
+ strict = d.getVar("BB_STRICT_CHECKSUM") or "0"
+
+ # If strict checking enabled and neither sum defined, raise error
+ if strict == "1":
+ raise NoChecksumError(checksum_line)
+
+ checksum_event = {"sha256sum": checksum}
+ bb.event.fire(MissingChecksumEvent(ud.url, **checksum_event), d)
+
+ if strict == "ignore":
+ return
+
+ # Log missing digest so user can more easily add it
+ logger.warning(
+ f"Missing checksum for '{ud.localpath}', consider using this " \
+ f"SRC_URI in the recipe:\n{checksum_line}")
+
+ def unpack(self, ud, rootdir, d):
+ image_file = ud.localname + ":" + ud.tag + ".zst"
+ oe.path.remove(rootdir + "/" + image_file)
+ oe.path.copyhardlink(ud.localpath, rootdir + "/" + image_file)
--
2.43.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/82da88bf02bf928d8807bc93bfb5fcdeece1f558.1721407122.git.jan.kiszka%40siemens.com.
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [PATCH v4 1/5] Introduce fetcher from container registries
2024-07-19 16:38 ` [PATCH v4 1/5] Introduce fetcher from container registries 'Jan Kiszka' via isar-users
@ 2024-07-25 10:48 ` 'Niedermayr, BENEDIKT' via isar-users
2024-07-25 11:10 ` 'Niedermayr, BENEDIKT' via isar-users
0 siblings, 1 reply; 12+ messages in thread
From: 'Niedermayr, BENEDIKT' via isar-users @ 2024-07-25 10:48 UTC (permalink / raw)
To: isar-users, Kiszka, Jan; +Cc: Cirujano Cuesta, Silvano, MOESSBAUER, Felix
On Fri, 2024-07-19 at 18:38 +0200, Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
>
> This bitbake fetcher allows to pull container images from registries,
> store them in the download cache and transfer them into the workdir of
> recipes requesting the image. The format of the URL is
>
> docker://[<host>/]<image>;digest=sha256:...[;tag=<tag>]
>
> Fetching without digest is supported but will cause a warning, just like
> downloading via wget without a checksum. If tag is left out, "latest" is
> used.
>
> In case a multi-arch image is specified, the fetcher will only pull for
> the package architecture of the requesting recipe. The image is stored
> compressed in docker-archive format and, wherever possible, hard-linked
> from DL_DIR to WORKDIR. Future versions may also introduce full
> unpacking of the fetched container layers in workdir if use cases come up.
>
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
> meta/classes/dpkg-base.bbclass | 6 +++
> meta/lib/container_fetcher.py | 86 ++++++++++++++++++++++++++++++++++
> 2 files changed, 92 insertions(+)
> create mode 100644 meta/lib/container_fetcher.py
>
> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
> index 789d6c74..d90b32a9 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -98,6 +98,12 @@ python() {
> if len(d.getVar('SRC_APT').strip()) > 0:
> bb.build.addtask('apt_unpack', 'do_patch', '', d)
> bb.build.addtask('cleanall_apt', 'do_cleanall', '', d)
> +
> + # container docker fetcher
> + import container_fetcher
> + from bb.fetch2 import methods
> +
> + methods.append(container_fetcher.Container())
> }
>
> do_apt_fetch() {
> diff --git a/meta/lib/container_fetcher.py b/meta/lib/container_fetcher.py
> new file mode 100644
> index 00000000..0d659154
> --- /dev/null
> +++ b/meta/lib/container_fetcher.py
> @@ -0,0 +1,86 @@
> +# This software is a part of ISAR.
> +# Copyright (c) Siemens AG, 2024
> +#
> +# SPDX-License-Identifier: MIT
> +
> +import oe.path
> +import os
> +import tempfile
> +from bb.fetch2 import FetchMethod
> +from bb.fetch2 import logger
> +from bb.fetch2 import MissingChecksumEvent
> +from bb.fetch2 import NoChecksumError
> +from bb.fetch2 import runfetchcmd
> +
> +class Container(FetchMethod):
> + def supports(self, ud, d):
> + return ud.type in ['docker']
> +
> + def urldata_init(self, ud, d):
> + ud.tag = "latest"
> + if "tag" in ud.parm:
> + ud.tag = ud.parm["tag"]
> +
> + ud.digest = None
> + if "digest" in ud.parm:
> + ud.digest = ud.parm["digest"]
> +
> + ud.arch = d.getVar('PACKAGE_ARCH')
> + ud.variant = None
> + if ud.arch == "armhf":
> + ud.arch = "arm"
> + ud.variant = "v7"
> + elif ud.arch == "armel":
> + ud.arch = "arm"
> + ud.variant = "v6"
> +
> + ud.container_name = ud.host + (ud.path if ud.path != "/" else "")
> + ud.container_src = ud.container_name + \
> + ("@" + ud.digest if ud.digest else ":" + ud.tag)
> + ud.localname = ud.container_name.replace('/', '.')
> + ud.localfile = "container-images/" + ud.arch + "/" + \
> + (ud.variant + "/" if ud.variant else "") + ud.localname + \
> + "_" + (ud.digest.replace(":", "-") if ud.digest else ud.tag) + \
> + ".zst"
> +
> + def download(self, ud, d):
> + tarball = ud.localfile[:-len('.zst')]
> + with tempfile.TemporaryDirectory(dir=d.getVar('DL_DIR')) as tmpdir:
> + # Take a two steps for downloading into a docker archive because
> + # not all source may have the required Docker schema 2 manifest.
> + runfetchcmd("skopeo copy --preserve-digests " + \
> + f"--override-arch {ud.arch} " + \
> + (f"--override-variant {ud.variant} " if ud.variant else "") + \
> + f"docker://{ud.container_src} dir:{tmpdir}", d)
> + runfetchcmd(f"skopeo copy dir:{tmpdir} " + \
> + f"docker-archive:{tarball}:{ud.container_name}:{ud.tag}", d)
> + zstd_defaults = d.getVar('ZSTD_DEFAULTS')
> + runfetchcmd(f"zstd -f --rm {zstd_defaults} {tarball}", d)
do_fetch doesn't get triggered if "ZSTD_LEVEL" changes, but it should, since the
output zst file changes with a different ZSTD_LEVEL.
Even adding a do_fetch[vardeps] += "ZSTD_DEFAULTS" doesn't lead to the desired behavior:
When ZSTD_LEVEL changes, the fetch task is run but does not actually fetch anything and
immediately proceeds with do_unpack. More precisely, the download() Method of the Container()
class is not getting executed, maybe due to some logic in the upper/surrounding Fetch() class.
That would be fine if the compression task didn't implement the compression...
Another thing to mention is that he whole fetch task is run again only if the compression changes.
Skopeo seems to override already downloaded layers rather than skipping them, which means that
changing the compression also means a complete new download process.
Maybe the unpack() task for packing and hardlinking the images would be better, even though it
sounds strange to put an compression task into the unpack task.
Benedikt
> +
> + if ud.digest:
> + return
> +
> + checksum = bb.utils.sha256_file(ud.localpath + "/manifest.json")
> + checksum_line = f"SRC_URI = \"{ud.url};digest=sha256:{checksum}\""
> +
> + strict = d.getVar("BB_STRICT_CHECKSUM") or "0"
> +
> + # If strict checking enabled and neither sum defined, raise error
> + if strict == "1":
> + raise NoChecksumError(checksum_line)
> +
> + checksum_event = {"sha256sum": checksum}
> + bb.event.fire(MissingChecksumEvent(ud.url, **checksum_event), d)
> +
> + if strict == "ignore":
> + return
> +
> + # Log missing digest so user can more easily add it
> + logger.warning(
> + f"Missing checksum for '{ud.localpath}', consider using this " \
> + f"SRC_URI in the recipe:\n{checksum_line}")
> +
> + def unpack(self, ud, rootdir, d):
> + image_file = ud.localname + ":" + ud.tag + ".zst"
> + oe.path.remove(rootdir + "/" + image_file)
> + oe.path.copyhardlink(ud.localpath, rootdir + "/" + image_file)
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/6beda53beb7a2cf672dca87757d4ca885cb62c93.camel%40siemens.com.
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [PATCH v4 1/5] Introduce fetcher from container registries
2024-07-25 10:48 ` 'Niedermayr, BENEDIKT' via isar-users
@ 2024-07-25 11:10 ` 'Niedermayr, BENEDIKT' via isar-users
0 siblings, 0 replies; 12+ messages in thread
From: 'Niedermayr, BENEDIKT' via isar-users @ 2024-07-25 11:10 UTC (permalink / raw)
To: isar-users, Kiszka, Jan; +Cc: Cirujano Cuesta, Silvano, MOESSBAUER, Felix
On Thu, 2024-07-25 at 10:48 +0000, 'Niedermayr, BENEDIKT' via isar-users wrote:
> On Fri, 2024-07-19 at 18:38 +0200, Jan Kiszka wrote:
> > From: Jan Kiszka <jan.kiszka@siemens.com>
> >
> > This bitbake fetcher allows to pull container images from registries,
> > store them in the download cache and transfer them into the workdir of
> > recipes requesting the image. The format of the URL is
> >
> > docker://[<host>/]<image>;digest=sha256:...[;tag=<tag>]
> >
> > Fetching without digest is supported but will cause a warning, just like
> > downloading via wget without a checksum. If tag is left out, "latest" is
> > used.
> >
> > In case a multi-arch image is specified, the fetcher will only pull for
> > the package architecture of the requesting recipe. The image is stored
> > compressed in docker-archive format and, wherever possible, hard-linked
> > from DL_DIR to WORKDIR. Future versions may also introduce full
> > unpacking of the fetched container layers in workdir if use cases come up.
> >
> > Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> > ---
> > meta/classes/dpkg-base.bbclass | 6 +++
> > meta/lib/container_fetcher.py | 86 ++++++++++++++++++++++++++++++++++
> > 2 files changed, 92 insertions(+)
> > create mode 100644 meta/lib/container_fetcher.py
> >
> > diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
> > index 789d6c74..d90b32a9 100644
> > --- a/meta/classes/dpkg-base.bbclass
> > +++ b/meta/classes/dpkg-base.bbclass
> > @@ -98,6 +98,12 @@ python() {
> > if len(d.getVar('SRC_APT').strip()) > 0:
> > bb.build.addtask('apt_unpack', 'do_patch', '', d)
> > bb.build.addtask('cleanall_apt', 'do_cleanall', '', d)
> > +
> > + # container docker fetcher
> > + import container_fetcher
> > + from bb.fetch2 import methods
> > +
> > + methods.append(container_fetcher.Container())
> > }
> >
> > do_apt_fetch() {
> > diff --git a/meta/lib/container_fetcher.py b/meta/lib/container_fetcher.py
> > new file mode 100644
> > index 00000000..0d659154
> > --- /dev/null
> > +++ b/meta/lib/container_fetcher.py
> > @@ -0,0 +1,86 @@
> > +# This software is a part of ISAR.
> > +# Copyright (c) Siemens AG, 2024
> > +#
> > +# SPDX-License-Identifier: MIT
> > +
> > +import oe.path
> > +import os
> > +import tempfile
> > +from bb.fetch2 import FetchMethod
> > +from bb.fetch2 import logger
> > +from bb.fetch2 import MissingChecksumEvent
> > +from bb.fetch2 import NoChecksumError
> > +from bb.fetch2 import runfetchcmd
> > +
> > +class Container(FetchMethod):
> > + def supports(self, ud, d):
> > + return ud.type in ['docker']
> > +
> > + def urldata_init(self, ud, d):
> > + ud.tag = "latest"
> > + if "tag" in ud.parm:
> > + ud.tag = ud.parm["tag"]
> > +
> > + ud.digest = None
> > + if "digest" in ud.parm:
> > + ud.digest = ud.parm["digest"]
> > +
> > + ud.arch = d.getVar('PACKAGE_ARCH')
> > + ud.variant = None
> > + if ud.arch == "armhf":
> > + ud.arch = "arm"
> > + ud.variant = "v7"
> > + elif ud.arch == "armel":
> > + ud.arch = "arm"
> > + ud.variant = "v6"
> > +
> > + ud.container_name = ud.host + (ud.path if ud.path != "/" else "")
> > + ud.container_src = ud.container_name + \
> > + ("@" + ud.digest if ud.digest else ":" + ud.tag)
> > + ud.localname = ud.container_name.replace('/', '.')
> > + ud.localfile = "container-images/" + ud.arch + "/" + \
> > + (ud.variant + "/" if ud.variant else "") + ud.localname + \
> > + "_" + (ud.digest.replace(":", "-") if ud.digest else ud.tag) + \
> > + ".zst"
> > +
> > + def download(self, ud, d):
> > + tarball = ud.localfile[:-len('.zst')]
> > + with tempfile.TemporaryDirectory(dir=d.getVar('DL_DIR')) as tmpdir:
> > + # Take a two steps for downloading into a docker archive because
> > + # not all source may have the required Docker schema 2 manifest.
> > + runfetchcmd("skopeo copy --preserve-digests " + \
> > + f"--override-arch {ud.arch} " + \
> > + (f"--override-variant {ud.variant} " if ud.variant else "") + \
> > + f"docker://{ud.container_src} dir:{tmpdir}", d)
> > + runfetchcmd(f"skopeo copy dir:{tmpdir} " + \
> > + f"docker-archive:{tarball}:{ud.container_name}:{ud.tag}", d)
> > + zstd_defaults = d.getVar('ZSTD_DEFAULTS')
> > + runfetchcmd(f"zstd -f --rm {zstd_defaults} {tarball}", d)
>
> do_fetch doesn't get triggered if "ZSTD_LEVEL" changes, but it should, since the
> output zst file changes with a different ZSTD_LEVEL.
>
> Even adding a do_fetch[vardeps] += "ZSTD_DEFAULTS" doesn't lead to the desired behavior:
>
> When ZSTD_LEVEL changes, the fetch task is run but does not actually fetch anything and
> immediately proceeds with do_unpack. More precisely, the download() Method of the Container()
> class is not getting executed, maybe due to some logic in the upper/surrounding Fetch() class.
> That would be fine if the compression task didn't implement the compression...
Sorry wrong wording:
That would be fine if the FETCH task didn't implement the compression...
Benedikt
>
> Another thing to mention is that he whole fetch task is run again only if the compression changes.
> Skopeo seems to override already downloaded layers rather than skipping them, which means that
> changing the compression also means a complete new download process.
>
> Maybe the unpack() task for packing and hardlinking the images would be better, even though it
> sounds strange to put an compression task into the unpack task.
>
> Benedikt
>
> > +
> > + if ud.digest:
> > + return
> > +
> > + checksum = bb.utils.sha256_file(ud.localpath + "/manifest.json")
> > + checksum_line = f"SRC_URI = \"{ud.url};digest=sha256:{checksum}\""
> > +
> > + strict = d.getVar("BB_STRICT_CHECKSUM") or "0"
> > +
> > + # If strict checking enabled and neither sum defined, raise error
> > + if strict == "1":
> > + raise NoChecksumError(checksum_line)
> > +
> > + checksum_event = {"sha256sum": checksum}
> > + bb.event.fire(MissingChecksumEvent(ud.url, **checksum_event), d)
> > +
> > + if strict == "ignore":
> > + return
> > +
> > + # Log missing digest so user can more easily add it
> > + logger.warning(
> > + f"Missing checksum for '{ud.localpath}', consider using this " \
> > + f"SRC_URI in the recipe:\n{checksum_line}")
> > +
> > + def unpack(self, ud, rootdir, d):
> > + image_file = ud.localname + ":" + ud.tag + ".zst"
> > + oe.path.remove(rootdir + "/" + image_file)
> > + oe.path.copyhardlink(ud.localpath, rootdir + "/" + image_file)
>
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/66a5c91e5afa23acba3f21270e59abebe82872ed.camel%40siemens.com.
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v4 2/5] container-loader: Introduce helper to load container images into local registry
2024-07-19 16:38 [PATCH v4 0/5] Introduce container fetcher and pre-loader 'Jan Kiszka' via isar-users
2024-07-19 16:38 ` [PATCH v4 1/5] Introduce fetcher from container registries 'Jan Kiszka' via isar-users
@ 2024-07-19 16:38 ` 'Jan Kiszka' via isar-users
2024-07-19 16:38 ` [PATCH v4 3/5] meta-isar: Add demo packages for installing prebuilt containers 'Jan Kiszka' via isar-users
` (3 subsequent siblings)
5 siblings, 0 replies; 12+ messages in thread
From: 'Jan Kiszka' via isar-users @ 2024-07-19 16:38 UTC (permalink / raw)
To: isar-users; +Cc: Silvano Cirujano-Cuesta, Benedikt Niedermayr, Felix Moessbauer
From: Jan Kiszka <jan.kiszka@siemens.com>
This allows to write dpkg-raw recipes which packages archived container
images and load them into a local docker or podman registry on boot. The
scenario behind this is to pre-fill local registries in a way that still
permits live updates during runtime.
The loader script only process images which are not yet available under
the same name and tag in the local registry. Also after loading, the
archived images stay on the local file system. This allows to perform
reloading in case the local registry should be emptied (e.g. reset to
factory state). To reduce the space those original images need, they are
compressed with zstd.
Separate include files are available to cater the main container
engines, one for docker and one for podman.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
.../container-loader/container-loader.inc | 73 +++++++++++++++++++
.../container-loader/docker-loader.inc | 10 +++
.../files/container-loader.service.tmpl | 12 +++
.../files/container-loader.sh.tmpl | 18 +++++
.../container-loader/podman-loader.inc | 10 +++
5 files changed, 123 insertions(+)
create mode 100644 meta/recipes-support/container-loader/container-loader.inc
create mode 100644 meta/recipes-support/container-loader/docker-loader.inc
create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl
create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl
create mode 100644 meta/recipes-support/container-loader/podman-loader.inc
diff --git a/meta/recipes-support/container-loader/container-loader.inc b/meta/recipes-support/container-loader/container-loader.inc
new file mode 100644
index 00000000..5fd8d23c
--- /dev/null
+++ b/meta/recipes-support/container-loader/container-loader.inc
@@ -0,0 +1,73 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+FILESPATH:append := ":${FILE_DIRNAME}/files"
+
+inherit dpkg-raw
+
+SRC_URI += " \
+ file://container-loader.service.tmpl \
+ file://container-loader.sh.tmpl"
+
+CONTAINER_DELETE_AFTER_LOAD ?= "0"
+
+DEBIAN_DEPENDS += "${CONTAINER_ENGINE_PACKAGES}, zstd"
+
+TEMPLATE_FILES += " \
+ container-loader.service.tmpl \
+ container-loader.sh.tmpl"
+TEMPLATE_VARS += " \
+ CONTAINER_ENGINE \
+ CONTAINER_DELETE_AFTER_LOAD"
+
+do_install() {
+ install -m 755 ${WORKDIR}/container-loader.sh ${D}/usr/share/${BPN}
+}
+do_install[cleandirs] += " \
+ ${D}/usr/share/${BPN} \
+ ${D}/usr/share/${BPN}/images"
+
+python do_install_fetched_containers() {
+ from oe.path import copyhardlink
+
+ workdir = d.getVar('WORKDIR')
+ D = d.getVar('D')
+ BPN = d.getVar('BPN')
+
+ image_list = open(D + "/usr/share/" + BPN + "/image.list", "w")
+
+ src_uri = d.getVar('SRC_URI').split()
+ for uri in src_uri:
+ scheme, host, path, _, _, parm = bb.fetch.decodeurl(uri)
+ if scheme != "docker":
+ continue
+
+ tag = parm["tag"] if "tag" in parm else "latest"
+ image_name = host + (path if path != "/" else "")
+ image_file = image_name.replace('/', '.') + \
+ ":" + tag + ".zst"
+ dest_dir = D + "/usr/share/" + BPN + "/images"
+
+ copyhardlink(workdir + "/" + image_file, dest_dir + "/" + image_file)
+
+ line = f"{image_file} {image_name}:{tag}"
+ bb.note(f"adding '{line}' to image.list")
+ image_list.write(line + "\n")
+
+ image_list.close()
+}
+
+addtask install_fetched_containers after do_install before do_prepare_build
+
+do_prepare_build:append() {
+ install -v -m 644 ${WORKDIR}/container-loader.service ${S}/debian/${BPN}.service
+
+ # Do not compress the package, most of its payload is already, and trying
+ # nevertheless will only cost time without any gain.
+ cat <<EOF >> ${S}/debian/rules
+override_dh_builddeb:
+ dh_builddeb -- -Znone
+EOF
+}
diff --git a/meta/recipes-support/container-loader/docker-loader.inc b/meta/recipes-support/container-loader/docker-loader.inc
new file mode 100644
index 00000000..b864c854
--- /dev/null
+++ b/meta/recipes-support/container-loader/docker-loader.inc
@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+require container-loader.inc
+
+CONTAINER_ENGINE = "docker"
+
+CONTAINER_ENGINE_PACKAGES ?= "docker.io, apparmor"
diff --git a/meta/recipes-support/container-loader/files/container-loader.service.tmpl b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
new file mode 100644
index 00000000..1638eaf2
--- /dev/null
+++ b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
@@ -0,0 +1,12 @@
+[Unit]
+Description=Load archived container images on boot
+After=${CONTAINER_ENGINE}.service
+Requires=${CONTAINER_ENGINE}.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/share/${BPN}/container-loader.sh
+RemainAfterExit=true
+
+[Install]
+WantedBy=multi-user.target
diff --git a/meta/recipes-support/container-loader/files/container-loader.sh.tmpl b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl
new file mode 100755
index 00000000..2356e31c
--- /dev/null
+++ b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl
@@ -0,0 +1,18 @@
+#!/bin/sh
+#
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+set -eu
+
+while read -r image ref; do
+ if [ -e /usr/share/${BPN}/images/"$image" ] && \
+ [ -z "$(${CONTAINER_ENGINE} images -q "$ref")" ]; then
+ pzstd -c -d /usr/share/${BPN}/images/"$image" | \
+ ${CONTAINER_ENGINE} load
+ if [ "${CONTAINER_DELETE_AFTER_LOAD}" = "1" ]; then
+ rm -f /usr/share/${BPN}/images/"$image"
+ fi
+ fi
+done < /usr/share/${BPN}/image.list
diff --git a/meta/recipes-support/container-loader/podman-loader.inc b/meta/recipes-support/container-loader/podman-loader.inc
new file mode 100644
index 00000000..d2c9a12d
--- /dev/null
+++ b/meta/recipes-support/container-loader/podman-loader.inc
@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+require container-loader.inc
+
+CONTAINER_ENGINE = "podman"
+
+CONTAINER_ENGINE_PACKAGES ?= "podman"
--
2.43.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/5af6163750f7ae0cb186e52727afe3ced1db2ce2.1721407122.git.jan.kiszka%40siemens.com.
^ permalink raw reply [flat|nested] 12+ messages in thread* [PATCH v4 3/5] meta-isar: Add demo packages for installing prebuilt containers
2024-07-19 16:38 [PATCH v4 0/5] Introduce container fetcher and pre-loader 'Jan Kiszka' via isar-users
2024-07-19 16:38 ` [PATCH v4 1/5] Introduce fetcher from container registries 'Jan Kiszka' via isar-users
2024-07-19 16:38 ` [PATCH v4 2/5] container-loader: Introduce helper to load container images into local registry 'Jan Kiszka' via isar-users
@ 2024-07-19 16:38 ` 'Jan Kiszka' via isar-users
2024-07-19 16:38 ` [PATCH v4 4/5] ci: Add test cases for container fetching and loading 'Jan Kiszka' via isar-users
` (2 subsequent siblings)
5 siblings, 0 replies; 12+ messages in thread
From: 'Jan Kiszka' via isar-users @ 2024-07-19 16:38 UTC (permalink / raw)
To: isar-users; +Cc: Silvano Cirujano-Cuesta, Benedikt Niedermayr, Felix Moessbauer
From: Jan Kiszka <jan.kiszka@siemens.com>
One recipe for docker, one for podman. Both pull from a registry that,
in contrast to infamous dockerhub, should not throttle CI jobs running
these frequently for testing purposes.
The podman variant of the recipe is intentionally leaving out the digest
to trigger the related warning of the container fetcher.
These demos also come with kas integration.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
kas/package/Kconfig | 19 +++++++++++++++++++
kas/package/pkg_prebuilt-docker-img.yaml | 9 +++++++++
kas/package/pkg_prebuilt-podman-img.yaml | 9 +++++++++
.../prebuilt-docker-img_0.1.bb | 12 ++++++++++++
.../prebuilt-podman-img_0.1.bb | 10 ++++++++++
5 files changed, 59 insertions(+)
create mode 100644 kas/package/pkg_prebuilt-docker-img.yaml
create mode 100644 kas/package/pkg_prebuilt-podman-img.yaml
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb
diff --git a/kas/package/Kconfig b/kas/package/Kconfig
index 35ba7cf1..395c3a08 100644
--- a/kas/package/Kconfig
+++ b/kas/package/Kconfig
@@ -146,6 +146,25 @@ config KAS_INCLUDE_PACKAGE_ISAR_CI_SSH_SETUP
default "kas/package/pkg_sshd-regen-keys.yaml"
depends on PACKAGE_ISAR_CI_SSH_SETUP
+
+config PACKAGE_PREBUILT_DOCKER_IMG
+ bool "prebuilt-docker-img"
+ default y
+
+config KAS_INCLUDE_PACKAGE_PREBUILT_DOCKER_IMG
+ string
+ default "kas/package/pkg_prebuilt-docker-img.yaml"
+ depends on PACKAGE_PREBUILT_DOCKER_IMG
+
+config PACKAGE_PREBUILT_PODMAN_IMG
+ bool "prebuilt-podman-img"
+ default y
+
+config KAS_INCLUDE_PACKAGE_PREBUILT_PODMAN_IMG
+ string
+ default "kas/package/pkg_prebuilt-podman-img.yaml"
+ depends on PACKAGE_PREBUILT_PODMAN_IMG
+
endmenu
config KAS_IMAGE_PREINSTALL
diff --git a/kas/package/pkg_prebuilt-docker-img.yaml b/kas/package/pkg_prebuilt-docker-img.yaml
new file mode 100644
index 00000000..df96a484
--- /dev/null
+++ b/kas/package/pkg_prebuilt-docker-img.yaml
@@ -0,0 +1,9 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+
+header:
+ version: 14
+
+local_conf_header:
+ package-prebuilt-docker-img: |
+ IMAGE_INSTALL:append = " prebuilt-docker-img"
diff --git a/kas/package/pkg_prebuilt-podman-img.yaml b/kas/package/pkg_prebuilt-podman-img.yaml
new file mode 100644
index 00000000..d0b8da1c
--- /dev/null
+++ b/kas/package/pkg_prebuilt-podman-img.yaml
@@ -0,0 +1,9 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+
+header:
+ version: 14
+
+local_conf_header:
+ package-prebuilt-podman-img: |
+ IMAGE_INSTALL:append = " prebuilt-podman-img"
diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
new file mode 100644
index 00000000..0dfc9b8f
--- /dev/null
+++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
@@ -0,0 +1,12 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+require recipes-support/container-loader/docker-loader.inc
+
+CONTAINER_DELETE_AFTER_LOAD = "1"
+
+SRC_URI += "\
+ docker://quay.io/libpod/alpine;digest=sha256:fa93b01658e3a5a1686dc3ae55f170d8de487006fb53a28efcd12ab0710a2e5f;tag=3.10.2 \
+ "
diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb
new file mode 100644
index 00000000..e671a494
--- /dev/null
+++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb
@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+require recipes-support/container-loader/podman-loader.inc
+
+SRC_URI += "\
+ docker://quay.io/libpod/alpine;tag=latest \
+ "
--
2.43.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/27ef7f39f79a72f188f78637f25ac8a616f0947a.1721407122.git.jan.kiszka%40siemens.com.
^ permalink raw reply [flat|nested] 12+ messages in thread* [PATCH v4 4/5] ci: Add test cases for container fetching and loading
2024-07-19 16:38 [PATCH v4 0/5] Introduce container fetcher and pre-loader 'Jan Kiszka' via isar-users
` (2 preceding siblings ...)
2024-07-19 16:38 ` [PATCH v4 3/5] meta-isar: Add demo packages for installing prebuilt containers 'Jan Kiszka' via isar-users
@ 2024-07-19 16:38 ` 'Jan Kiszka' via isar-users
2024-07-30 7:44 ` Uladzimir Bely
2024-07-19 16:38 ` [PATCH v4 5/5] doc: Describe how to use the container fetcher and loader 'Jan Kiszka' via isar-users
2024-08-08 6:35 ` [PATCH v4 0/5] Introduce container fetcher and pre-loader Uladzimir Bely
5 siblings, 1 reply; 12+ messages in thread
From: 'Jan Kiszka' via isar-users @ 2024-07-19 16:38 UTC (permalink / raw)
To: isar-users; +Cc: Silvano Cirujano-Cuesta, Benedikt Niedermayr, Felix Moessbauer
From: Jan Kiszka <jan.kiszka@siemens.com>
This plugs the two example recipes for loading container images into
VM-based testing. The test consists of running 'true' in the installed
alpine images.
Rather than enabling the ci user to do password-less sudo, this uses su
with the piped-in password. Another trick needed is to poll for the
images because loading is performed asynchronously.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
.../recipes-core/images/isar-image-ci.bb | 2 ++
testsuite/citest.py | 21 +++++++++++++++++++
2 files changed, 23 insertions(+)
diff --git a/meta-test/recipes-core/images/isar-image-ci.bb b/meta-test/recipes-core/images/isar-image-ci.bb
index e5d51e6e..9133da74 100644
--- a/meta-test/recipes-core/images/isar-image-ci.bb
+++ b/meta-test/recipes-core/images/isar-image-ci.bb
@@ -16,6 +16,7 @@ IMAGE_INSTALL += "sshd-regen-keys"
# qemuamd64-bookworm
WKS_FILE:qemuamd64:debian-bookworm ?= "multipart-efi.wks"
+IMAGE_INSTALL:append:qemuamd64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img"
# qemuamd64-bullseye
IMAGE_FSTYPES:append:qemuamd64:debian-bullseye ?= " cpio.gz tar.gz"
@@ -51,3 +52,4 @@ IMAGER_INSTALL:append:qemuarm:debian-bookworm ?= " ${SYSTEMD_BOOTLOADER_INSTALL}
# qemuarm64-bookworm
IMAGE_FSTYPES:append:qemuarm64:debian-bookworm ?= " wic.xz"
IMAGER_INSTALL:append:qemuarm64:debian-bookworm ?= " ${GRUB_BOOTLOADER_INSTALL}"
+IMAGE_INSTALL:append:qemuarm64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img"
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 8dd907d0..539c9440 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -522,3 +522,24 @@ class VmBootTestFull(CIBaseTest):
self.init()
self.vm_start('mipsel','bookworm', image='isar-image-ci',
script='test_kernel_module.sh example_module')
+
+
+ def test_amd64_bookworm_prebuilt_containers(self):
+ self.init()
+ self.vm_start('amd64', 'bookworm', image='isar-image-ci',
+ cmd='echo root | su -c \'' \
+ 'PATH=\$PATH:/usr/sbin;' \
+ 'for n in \$(seq 30); do docker images | grep -q alpine && break; sleep 10; done;' \
+ 'docker run --rm quay.io/libpod/alpine:3.10.2 true && ' \
+ 'for n in \$(seq 30); do podman images | grep -q alpine && break; sleep 10; done;' \
+ 'podman run --rm quay.io/libpod/alpine:latest true\'')
+
+ def test_arm64_bookworm_prebuilt_containers(self):
+ self.init()
+ self.vm_start('arm64', 'bookworm', image='isar-image-ci',
+ cmd='echo root | su -c \'' \
+ 'PATH=\$PATH:/usr/sbin;' \
+ 'for n in \$(seq 30); do docker images | grep -q alpine && break; sleep 10; done;' \
+ 'docker run --rm quay.io/libpod/alpine:3.10.2 true && ' \
+ 'for n in \$(seq 30); do podman images | grep -q alpine && break; sleep 10; done;' \
+ 'podman run --rm quay.io/libpod/alpine:latest true\'')
--
2.43.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/3a0b76b54aca9cee33d249a2df73d233edf08708.1721407122.git.jan.kiszka%40siemens.com.
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [PATCH v4 4/5] ci: Add test cases for container fetching and loading
2024-07-19 16:38 ` [PATCH v4 4/5] ci: Add test cases for container fetching and loading 'Jan Kiszka' via isar-users
@ 2024-07-30 7:44 ` Uladzimir Bely
2024-07-30 21:45 ` 'Jan Kiszka' via isar-users
0 siblings, 1 reply; 12+ messages in thread
From: Uladzimir Bely @ 2024-07-30 7:44 UTC (permalink / raw)
To: Jan Kiszka, isar-users
On Fri, 2024-07-19 at 18:38 +0200, 'Jan Kiszka' via isar-users wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
>
> This plugs the two example recipes for loading container images into
> VM-based testing. The test consists of running 'true' in the
> installed
> alpine images.
>
> Rather than enabling the ci user to do password-less sudo, this uses
> su
> with the piped-in password. Another trick needed is to poll for the
> images because loading is performed asynchronously.
>
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
> .../recipes-core/images/isar-image-ci.bb | 2 ++
> testsuite/citest.py | 21
> +++++++++++++++++++
> 2 files changed, 23 insertions(+)
>
> diff --git a/meta-test/recipes-core/images/isar-image-ci.bb b/meta-
> test/recipes-core/images/isar-image-ci.bb
> index e5d51e6e..9133da74 100644
> --- a/meta-test/recipes-core/images/isar-image-ci.bb
> +++ b/meta-test/recipes-core/images/isar-image-ci.bb
> @@ -16,6 +16,7 @@ IMAGE_INSTALL += "sshd-regen-keys"
>
> # qemuamd64-bookworm
> WKS_FILE:qemuamd64:debian-bookworm ?= "multipart-efi.wks"
> +IMAGE_INSTALL:append:qemuamd64:debian-bookworm = " prebuilt-docker-
> img prebuilt-podman-img"
>
> # qemuamd64-bullseye
> IMAGE_FSTYPES:append:qemuamd64:debian-bullseye ?= " cpio.gz tar.gz"
> @@ -51,3 +52,4 @@ IMAGER_INSTALL:append:qemuarm:debian-bookworm ?= "
> ${SYSTEMD_BOOTLOADER_INSTALL}
> # qemuarm64-bookworm
> IMAGE_FSTYPES:append:qemuarm64:debian-bookworm ?= " wic.xz"
> IMAGER_INSTALL:append:qemuarm64:debian-bookworm ?= "
> ${GRUB_BOOTLOADER_INSTALL}"
> +IMAGE_INSTALL:append:qemuarm64:debian-bookworm = " prebuilt-docker-
> img prebuilt-podman-img"
> diff --git a/testsuite/citest.py b/testsuite/citest.py
> index 8dd907d0..539c9440 100755
> --- a/testsuite/citest.py
> +++ b/testsuite/citest.py
> @@ -522,3 +522,24 @@ class VmBootTestFull(CIBaseTest):
> self.init()
> self.vm_start('mipsel','bookworm', image='isar-image-ci',
> script='test_kernel_module.sh example_module')
> +
> +
> + def test_amd64_bookworm_prebuilt_containers(self):
> + self.init()
> + self.vm_start('amd64', 'bookworm', image='isar-image-ci',
> + cmd='echo root | su -c \'' \
> + 'PATH=\$PATH:/usr/sbin;' \
> + 'for n in \$(seq 30); do docker images |
> grep -q alpine && break; sleep 10; done;' \
> + 'docker run --rm
> quay.io/libpod/alpine:3.10.2 true && ' \
> + 'for n in \$(seq 30); do podman images |
> grep -q alpine && break; sleep 10; done;' \
> + 'podman run --rm
> quay.io/libpod/alpine:latest true\'')
> +
> + def test_arm64_bookworm_prebuilt_containers(self):
> + self.init()
> + self.vm_start('arm64', 'bookworm', image='isar-image-ci',
> + cmd='echo root | su -c \'' \
> + 'PATH=\$PATH:/usr/sbin;' \
> + 'for n in \$(seq 30); do docker images |
> grep -q alpine && break; sleep 10; done;' \
> + 'docker run --rm
> quay.io/libpod/alpine:3.10.2 true && ' \
> + 'for n in \$(seq 30); do podman images |
> grep -q alpine && break; sleep 10; done;' \
> + 'podman run --rm
> quay.io/libpod/alpine:latest true\'')
This appears to fail in CI:
[stdlog] 2024-07-30 07:29:37,721 avocado.app ERROR| qemu-system-
aarch64: -drive
file=/workspace/build/isar_ub_devel/49/build/tmp/deploy/images/qemuarm6
4/isar-image-ci-debian-bookworm-
qemuarm64.ext4,if=none,format=raw,id=hd0: Could not open
'/workspace/build/isar_ub_devel/49/build/tmp/deploy/images/qemuarm64/is
ar-image-ci-debian-bookworm-qemuarm64.ext4': No such file or directory
Should be either changed the bullseye here (not yet checked if it
works). Or, we need to build arm64-bullseye image in build tests
before.
> --
> 2.43.0
>
--
Best regards,
Uladzimir.
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/624dbb193e9bfbaedbfb7e288b6b295bd9fe5a06.camel%40ilbers.de.
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [PATCH v4 4/5] ci: Add test cases for container fetching and loading
2024-07-30 7:44 ` Uladzimir Bely
@ 2024-07-30 21:45 ` 'Jan Kiszka' via isar-users
2024-08-05 7:20 ` Uladzimir Bely
0 siblings, 1 reply; 12+ messages in thread
From: 'Jan Kiszka' via isar-users @ 2024-07-30 21:45 UTC (permalink / raw)
To: Uladzimir Bely, isar-users
On 30.07.24 09:44, Uladzimir Bely wrote:
> On Fri, 2024-07-19 at 18:38 +0200, 'Jan Kiszka' via isar-users wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> This plugs the two example recipes for loading container images into
>> VM-based testing. The test consists of running 'true' in the
>> installed
>> alpine images.
>>
>> Rather than enabling the ci user to do password-less sudo, this uses
>> su
>> with the piped-in password. Another trick needed is to poll for the
>> images because loading is performed asynchronously.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>> .../recipes-core/images/isar-image-ci.bb | 2 ++
>> testsuite/citest.py | 21
>> +++++++++++++++++++
>> 2 files changed, 23 insertions(+)
>>
>> diff --git a/meta-test/recipes-core/images/isar-image-ci.bb b/meta-
>> test/recipes-core/images/isar-image-ci.bb
>> index e5d51e6e..9133da74 100644
>> --- a/meta-test/recipes-core/images/isar-image-ci.bb
>> +++ b/meta-test/recipes-core/images/isar-image-ci.bb
>> @@ -16,6 +16,7 @@ IMAGE_INSTALL += "sshd-regen-keys"
>>
>> # qemuamd64-bookworm
>> WKS_FILE:qemuamd64:debian-bookworm ?= "multipart-efi.wks"
>> +IMAGE_INSTALL:append:qemuamd64:debian-bookworm = " prebuilt-docker-
>> img prebuilt-podman-img"
>>
>> # qemuamd64-bullseye
>> IMAGE_FSTYPES:append:qemuamd64:debian-bullseye ?= " cpio.gz tar.gz"
>> @@ -51,3 +52,4 @@ IMAGER_INSTALL:append:qemuarm:debian-bookworm ?= "
>> ${SYSTEMD_BOOTLOADER_INSTALL}
>> # qemuarm64-bookworm
>> IMAGE_FSTYPES:append:qemuarm64:debian-bookworm ?= " wic.xz"
>> IMAGER_INSTALL:append:qemuarm64:debian-bookworm ?= "
>> ${GRUB_BOOTLOADER_INSTALL}"
>> +IMAGE_INSTALL:append:qemuarm64:debian-bookworm = " prebuilt-docker-
>> img prebuilt-podman-img"
>> diff --git a/testsuite/citest.py b/testsuite/citest.py
>> index 8dd907d0..539c9440 100755
>> --- a/testsuite/citest.py
>> +++ b/testsuite/citest.py
>> @@ -522,3 +522,24 @@ class VmBootTestFull(CIBaseTest):
>> self.init()
>> self.vm_start('mipsel','bookworm', image='isar-image-ci',
>> script='test_kernel_module.sh example_module')
>> +
>> +
>> + def test_amd64_bookworm_prebuilt_containers(self):
>> + self.init()
>> + self.vm_start('amd64', 'bookworm', image='isar-image-ci',
>> + cmd='echo root | su -c \'' \
>> + 'PATH=\$PATH:/usr/sbin;' \
>> + 'for n in \$(seq 30); do docker images |
>> grep -q alpine && break; sleep 10; done;' \
>> + 'docker run --rm
>> quay.io/libpod/alpine:3.10.2 true && ' \
>> + 'for n in \$(seq 30); do podman images |
>> grep -q alpine && break; sleep 10; done;' \
>> + 'podman run --rm
>> quay.io/libpod/alpine:latest true\'')
>> +
>> + def test_arm64_bookworm_prebuilt_containers(self):
>> + self.init()
>> + self.vm_start('arm64', 'bookworm', image='isar-image-ci',
>> + cmd='echo root | su -c \'' \
>> + 'PATH=\$PATH:/usr/sbin;' \
>> + 'for n in \$(seq 30); do docker images |
>> grep -q alpine && break; sleep 10; done;' \
>> + 'docker run --rm
>> quay.io/libpod/alpine:3.10.2 true && ' \
>> + 'for n in \$(seq 30); do podman images |
>> grep -q alpine && break; sleep 10; done;' \
>> + 'podman run --rm
>> quay.io/libpod/alpine:latest true\'')
>
> This appears to fail in CI:
>
> [stdlog] 2024-07-30 07:29:37,721 avocado.app ERROR| qemu-system-
> aarch64: -drive
> file=/workspace/build/isar_ub_devel/49/build/tmp/deploy/images/qemuarm6
> 4/isar-image-ci-debian-bookworm-
> qemuarm64.ext4,if=none,format=raw,id=hd0: Could not open
> '/workspace/build/isar_ub_devel/49/build/tmp/deploy/images/qemuarm64/is
> ar-image-ci-debian-bookworm-qemuarm64.ext4': No such file or directory
>
> Should be either changed the bullseye here (not yet checked if it
> works). Or, we need to build arm64-bullseye image in build tests
> before.
bullseye is legacy.
Unfortunately, the testsuite lacks a dependency model. Here,
VmBootTestFull depends on CrossTest and NoCrossTest - apparently this
was not the case so far.
I'm still wondering why we are using such an unhandy and non-standard
test framework if not even such basic things are expressed - or are even
expressible?
Jan
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/38ba708f-aacd-4f9b-a777-40a993335cc8%40web.de.
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [PATCH v4 4/5] ci: Add test cases for container fetching and loading
2024-07-30 21:45 ` 'Jan Kiszka' via isar-users
@ 2024-08-05 7:20 ` Uladzimir Bely
0 siblings, 0 replies; 12+ messages in thread
From: Uladzimir Bely @ 2024-08-05 7:20 UTC (permalink / raw)
To: Jan Kiszka, isar-users
On Tue, 2024-07-30 at 23:45 +0200, Jan Kiszka wrote:
> On 30.07.24 09:44, Uladzimir Bely wrote:
> > On Fri, 2024-07-19 at 18:38 +0200, 'Jan Kiszka' via isar-users
> > wrote:
> > > From: Jan Kiszka <jan.kiszka@siemens.com>
> > >
> > > This plugs the two example recipes for loading container images
> > > into
> > > VM-based testing. The test consists of running 'true' in the
> > > installed
> > > alpine images.
> > >
> > > Rather than enabling the ci user to do password-less sudo, this
> > > uses
> > > su
> > > with the piped-in password. Another trick needed is to poll for
> > > the
> > > images because loading is performed asynchronously.
> > >
> > > Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> > > ---
> > > .../recipes-core/images/isar-image-ci.bb | 2 ++
> > > testsuite/citest.py | 21
> > > +++++++++++++++++++
> > > 2 files changed, 23 insertions(+)
> > >
> > > diff --git a/meta-test/recipes-core/images/isar-image-ci.bb
> > > b/meta-
> > > test/recipes-core/images/isar-image-ci.bb
> > > index e5d51e6e..9133da74 100644
> > > --- a/meta-test/recipes-core/images/isar-image-ci.bb
> > > +++ b/meta-test/recipes-core/images/isar-image-ci.bb
> > > @@ -16,6 +16,7 @@ IMAGE_INSTALL += "sshd-regen-keys"
> > >
> > > # qemuamd64-bookworm
> > > WKS_FILE:qemuamd64:debian-bookworm ?= "multipart-efi.wks"
> > > +IMAGE_INSTALL:append:qemuamd64:debian-bookworm = " prebuilt-
> > > docker-
> > > img prebuilt-podman-img"
> > >
> > > # qemuamd64-bullseye
> > > IMAGE_FSTYPES:append:qemuamd64:debian-bullseye ?= " cpio.gz
> > > tar.gz"
> > > @@ -51,3 +52,4 @@ IMAGER_INSTALL:append:qemuarm:debian-bookworm
> > > ?= "
> > > ${SYSTEMD_BOOTLOADER_INSTALL}
> > > # qemuarm64-bookworm
> > > IMAGE_FSTYPES:append:qemuarm64:debian-bookworm ?= " wic.xz"
> > > IMAGER_INSTALL:append:qemuarm64:debian-bookworm ?= "
> > > ${GRUB_BOOTLOADER_INSTALL}"
> > > +IMAGE_INSTALL:append:qemuarm64:debian-bookworm = " prebuilt-
> > > docker-
> > > img prebuilt-podman-img"
> > > diff --git a/testsuite/citest.py b/testsuite/citest.py
> > > index 8dd907d0..539c9440 100755
> > > --- a/testsuite/citest.py
> > > +++ b/testsuite/citest.py
> > > @@ -522,3 +522,24 @@ class VmBootTestFull(CIBaseTest):
> > > self.init()
> > > self.vm_start('mipsel','bookworm', image='isar-image-
> > > ci',
> > > script='test_kernel_module.sh
> > > example_module')
> > > +
> > > +
> > > + def test_amd64_bookworm_prebuilt_containers(self):
> > > + self.init()
> > > + self.vm_start('amd64', 'bookworm', image='isar-image-
> > > ci',
> > > + cmd='echo root | su -c \'' \
> > > + 'PATH=\$PATH:/usr/sbin;' \
> > > + 'for n in \$(seq 30); do docker images
> > > |
> > > grep -q alpine && break; sleep 10; done;' \
> > > + 'docker run --rm
> > > quay.io/libpod/alpine:3.10.2 true && ' \
> > > + 'for n in \$(seq 30); do podman images
> > > |
> > > grep -q alpine && break; sleep 10; done;' \
> > > + 'podman run --rm
> > > quay.io/libpod/alpine:latest true\'')
> > > +
> > > + def test_arm64_bookworm_prebuilt_containers(self):
> > > + self.init()
> > > + self.vm_start('arm64', 'bookworm', image='isar-image-
> > > ci',
> > > + cmd='echo root | su -c \'' \
> > > + 'PATH=\$PATH:/usr/sbin;' \
> > > + 'for n in \$(seq 30); do docker images
> > > |
> > > grep -q alpine && break; sleep 10; done;' \
> > > + 'docker run --rm
> > > quay.io/libpod/alpine:3.10.2 true && ' \
> > > + 'for n in \$(seq 30); do podman images
> > > |
> > > grep -q alpine && break; sleep 10; done;' \
> > > + 'podman run --rm
> > > quay.io/libpod/alpine:latest true\'')
> >
> > This appears to fail in CI:
> >
> > [stdlog] 2024-07-30 07:29:37,721 avocado.app ERROR| qemu-system-
> > aarch64: -drive
> > file=/workspace/build/isar_ub_devel/49/build/tmp/deploy/images/qemu
> > arm6
> > 4/isar-image-ci-debian-bookworm-
> > qemuarm64.ext4,if=none,format=raw,id=hd0: Could not open
> > '/workspace/build/isar_ub_devel/49/build/tmp/deploy/images/qemuarm6
> > 4/is
> > ar-image-ci-debian-bookworm-qemuarm64.ext4': No such file or
> > directory
> >
> > Should be either changed the bullseye here (not yet checked if it
> > works). Or, we need to build arm64-bullseye image in build tests
> > before.
>
> bullseye is legacy.
>
Hello.
I rebased the patchset on top of "[PATCH] testsuite: Build bookworm
arm64 image in no-cross test". So, now this test doesn't fail this way.
But there were other issues related to the backslashed '\$'. I couldn't
catch this until applied one more useful patch: "[PATCH] testsuite:
Save ssh command stdout/stderr in case of failure" that allowed to see
syntax error here.
Also, flake8 appeared to recognize this as "W605 invalid escape
sequence '\$'".
I've just sent a replacement of the patch that fixes the issue and
python style here.
You could prepare pathset v5, or I could merge it as is after testing.
> Unfortunately, the testsuite lacks a dependency model. Here,
> VmBootTestFull depends on CrossTest and NoCrossTest - apparently this
> was not the case so far.
>
> I'm still wondering why we are using such an unhandy and non-standard
> test framework if not even such basic things are expressed - or are
> even
> expressible?
>
> Jan
>
--
Best regards,
Uladzimir.
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/ec827283af69b6564ae5915ada6e9def88f7ec6c.camel%40ilbers.de.
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v4 5/5] doc: Describe how to use the container fetcher and loader
2024-07-19 16:38 [PATCH v4 0/5] Introduce container fetcher and pre-loader 'Jan Kiszka' via isar-users
` (3 preceding siblings ...)
2024-07-19 16:38 ` [PATCH v4 4/5] ci: Add test cases for container fetching and loading 'Jan Kiszka' via isar-users
@ 2024-07-19 16:38 ` 'Jan Kiszka' via isar-users
2024-08-08 6:35 ` [PATCH v4 0/5] Introduce container fetcher and pre-loader Uladzimir Bely
5 siblings, 0 replies; 12+ messages in thread
From: 'Jan Kiszka' via isar-users @ 2024-07-19 16:38 UTC (permalink / raw)
To: isar-users; +Cc: Silvano Cirujano-Cuesta, Benedikt Niedermayr, Felix Moessbauer
From: Jan Kiszka <jan.kiszka@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
doc/user_manual.md | 58 ++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 58 insertions(+)
diff --git a/doc/user_manual.md b/doc/user_manual.md
index 776ae52c..2bdacbec 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -1519,3 +1519,61 @@ SBUILD_CHROOT_PREINSTALL_EXTRA += "<base packages>"
Then, in the dpkg recipe of your package, simply set `SBUILD_FLAVOR = "<your flavor>"`.
To install additional packages into the sbuild chroot, add them to `SBUILD_CHROOT_PREINSTALL_EXTRA`.
+
+## Pre-install container images
+
+If an isar-generated image shall provide a container runtime, it may also be
+desirable to pre-install container images to avoid having to download them on
+first boot or because they may not be accessible outside of the build
+environment. Isar supports this scenario via two services, a container fetcher
+and a container loader.
+
+### Bitbake fetcher for containers
+
+The bitbake fetching protocol "docker://" allows to download pre-built images
+from container registries. The URL consists of the image path, followed by
+a recommended digest in the form `digest=sha256:<sha256sum>` and an optional
+tag in the form `tag=<tag>`. A digest is preferred over a tag to identify an
+image when fetching because it also allows to validate its integrity. If a tag
+is not specified, `latest` is used as tag name.
+
+In case a multi-arch image is specified, the fetcher will only pull for the
+package architecture of the requesting recipe (`PACKAGE_ARCH`). The fetched
+images are stored as zstd-compressed in docker-archive format in the
+`WORKDIR` of the recipe. The name of the image is derived from the container
+image name, replacing all `/` with `.` and appending `:<tag>.zst`. Example:
+`docker://debian;tag=bookworm` will be saved as `debian:bookworm.zst`.
+
+### Container loader helpers
+
+To create a Debian package which can carry container images and load them into
+local storage of docker or podman, there is a set of helpers available. To use
+them in an own recipe, add
+`require recipes-support/container-loader/docker-loader.inc` when using docker
+and `require recipes-support/container-loader/podman-loader.inc` when using
+podman. The loader will try to transfer the packaged image into the container
+runtime storage on boot, but only if no container image of the same name and
+tag is present already.
+
+Unless `CONTAINER_DELETE_AFTER_LOAD` is set to `1`, the source container images
+remain by default available and may be used again for loading the storage after
+it may have been emptied later on (factory reset).
+
+Source container images may either be fetched as binaries from a registry, see
+above, or built via isar as well.
+
+### Example
+
+This creates a debian package which will download, package and then load the
+`debian:bookworm-20240701-slim` container image into the docker container
+storage. The package will depend on `docker.io`, insuring that that basic
+runtime services are installed on the target as well. The packaged image will
+be deleted from the target device's rootfs after successful import.
+
+```
+require recipes-support/container-loader/docker-loader.inc
+
+CONTAINER_DELETE_AFTER_LOAD = "1"
+
+SRC_URI += "docker://debian;digest=sha256:f528891ab1aa484bf7233dbcc84f3c806c3e427571d75510a9d74bb5ec535b33;tag=bookworm-20240701-slim"
+```
--
2.43.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/049353103858d43105d45603619a7548f2a29579.1721407122.git.jan.kiszka%40siemens.com.
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [PATCH v4 0/5] Introduce container fetcher and pre-loader
2024-07-19 16:38 [PATCH v4 0/5] Introduce container fetcher and pre-loader 'Jan Kiszka' via isar-users
` (4 preceding siblings ...)
2024-07-19 16:38 ` [PATCH v4 5/5] doc: Describe how to use the container fetcher and loader 'Jan Kiszka' via isar-users
@ 2024-08-08 6:35 ` Uladzimir Bely
5 siblings, 0 replies; 12+ messages in thread
From: Uladzimir Bely @ 2024-08-08 6:35 UTC (permalink / raw)
To: Jan Kiszka, isar-users
On Fri, 2024-07-19 at 18:38 +0200, 'Jan Kiszka' via isar-users wrote:
> We noticed internally that there are at least 3 implementations for
> solving the task of pre-loading container images into isar-built base
> systems, may it be that the images are not publicly available or the
> device is only poorly connected, if at all. This series tries to
> solve
> the task generically - hopefully.
>
> Changes in v4
> - fetch per architecture, no longer full multi-arch
> - directly archive and compress in download task
> - use hard links where possible to reduce disk usage
> - always use zstd, remove configurability
>
> Changes in v3:
> - do not compress container deb packages [Benedikt]
> - fix various spelling issues in the doc [Adriaan, Benedikt]
>
> Changes in v2:
> - switch to open-coded unpack before loading
> - switch to zstd as default compression
> - use default compression settings from bitbake.conf
> - add support for delete-after-load (opt-in)
> - factor out fetched container installation into separate task
> (should make self-built container packaging easier)
> - use BPN instead of PN
> - add "Requires" to systemd service
> - use PACKAGE_ARCH to select container arch (rather than
> DISTRO_ARCH)
> - add CI test
> - add documentation
>
> Jan
>
> Jan Kiszka (5):
> Introduce fetcher from container registries
> container-loader: Introduce helper to load container images into
> local
> registry
> meta-isar: Add demo packages for installing prebuilt containers
> ci: Add test cases for container fetching and loading
> doc: Describe how to use the container fetcher and loader
>
> doc/user_manual.md | 58 +++++++++++++
> kas/package/Kconfig | 19 ++++
> kas/package/pkg_prebuilt-docker-img.yaml | 9 ++
> kas/package/pkg_prebuilt-podman-img.yaml | 9 ++
> .../prebuilt-docker-img_0.1.bb | 12 +++
> .../prebuilt-podman-img_0.1.bb | 10 +++
> .../recipes-core/images/isar-image-ci.bb | 2 +
> meta/classes/dpkg-base.bbclass | 6 ++
> meta/lib/container_fetcher.py | 86
> +++++++++++++++++++
> .../container-loader/container-loader.inc | 73 ++++++++++++++++
> .../container-loader/docker-loader.inc | 10 +++
> .../files/container-loader.service.tmpl | 12 +++
> .../files/container-loader.sh.tmpl | 18 ++++
> .../container-loader/podman-loader.inc | 10 +++
> testsuite/citest.py | 21 +++++
> 15 files changed, 355 insertions(+)
> create mode 100644 kas/package/pkg_prebuilt-docker-img.yaml
> create mode 100644 kas/package/pkg_prebuilt-podman-img.yaml
> create mode 100644 meta-isar/recipes-app/prebuilt-
> container/prebuilt-docker-img_0.1.bb
> create mode 100644 meta-isar/recipes-app/prebuilt-
> container/prebuilt-podman-img_0.1.bb
> create mode 100644 meta/lib/container_fetcher.py
> create mode 100644 meta/recipes-support/container-loader/container-
> loader.inc
> create mode 100644 meta/recipes-support/container-loader/docker-
> loader.inc
> create mode 100644 meta/recipes-support/container-
> loader/files/container-loader.service.tmpl
> create mode 100755 meta/recipes-support/container-
> loader/files/container-loader.sh.tmpl
> create mode 100644 meta/recipes-support/container-loader/podman-
> loader.inc
>
> --
> 2.43.0
>
Applied to next (w/o patch 4 for CI), thanks.
--
Best regards,
Uladzimir.
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/2b56490fcf23aff01adcc3b69f3dc9c3a0c48fa5.camel%40ilbers.de.
^ permalink raw reply [flat|nested] 12+ messages in thread