From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Tue, 16 Jul 2024 16:18:20 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-ed1-f59.google.com (mail-ed1-f59.google.com [209.85.208.59]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46GEIJjt013323 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Jul 2024 16:18:20 +0200 Received: by mail-ed1-f59.google.com with SMTP id 4fb4d7f45d1cf-58cdd86c091sf4935001a12.3 for ; Tue, 16 Jul 2024 07:18:20 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721139494; cv=pass; d=google.com; s=arc-20160816; b=dqYYbb5yce8l0eVy+B2vmTeO1Q+7HhJeg+49FsMvIgHUwx7ixK/s9u90RBL0eJMjK/ KgLBhkni2KL8fRzbq4FLeAA71CGAxu0bh/2QHeARj5YZKpCY8aUfYKarhmphcnBHOZ0L u9QJ4szxPGDXWfNrQYkhDzm5wBbQ+4JwsNQKRmUxO0J8nx7LfVZ3hGFJ/W74p/1p8eFu ricptPfcdTIn9YifxRznbv0w+8D626o64Ki1I139P227tnR+gvf2D930t4zlw00lDXbQ FE0gH1UP7pqL1+ZD/zNlCxDHLXnEsKb6XslJ+T1dCJuq/Idz531v49xqtkW1Dh+VoXQ1 R5Rg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=WlfjU1+uJOmDYTuGwaL5xHA70bWDs/47rYtMBUB2API=; fh=qH24Pv2Xczh56u+mmvvP69NRDCyBiC7QOK6lm0ycY+w=; b=MnRo9eZk433hJEw9gv7jpNv+Q8d+zVjfj6H1kZPzF/WdMfE2HwaPNFvq+KB8GFSdgu 5u9SjJ15aXMPuymv+zczow25H3c0QhfNCa5gSxt6yDjE0SSdr/M9J8by6qfnpg9TQ9fJ QNNENzAl4ar3UEoBo0RrOajAL9nxPC5iq5KhLX3pJzlweTIi7G63MlLIznq1DvyHWpEA TPcjLQCH8QTTwDST8nV8oryDtYCwALK8EivrtoyR0j+bHv7axjfmAMXOil8pinG+Tfg8 oeRZtqvVRV9/08NJHkJcC8K50NmV0nyAj9QCvQCTbwHxmq8FX7mSqj+Q51XFc7gF4nVR QC7Q==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=BFfPEadz; spf=pass (google.com: domain of fm-294854-202407161418115ce2812f9612c0cc2e-xq_fc2@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-294854-202407161418115ce2812f9612c0cc2e-xq_fc2@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721139494; x=1721744294; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=WlfjU1+uJOmDYTuGwaL5xHA70bWDs/47rYtMBUB2API=; b=o0MQQjPmBLtOu7FoXGheP53bMPGc80xV8/YkU5673wBRkKiZdODr70TVLo+SZkrLL5 tks74xDyqdpR6BWZ0Q0Z/Z0aQYNk+gE59dXszpjgrVufB+6gpDir0hxFvGjp/t4Hf0Jo l0gtOTQpIalycA4HKOb2xiebnLw9wjRsFkoixTRQqQ8UifcI2SfFj2GebHj72RoAodDx a6XFlSejnYJQ3hs4OuIrRDzA67xeXm0boqDq3l0/cfb0OfWAi5gbpYSV/wWMaGP5MRcz rB5nbZ1qw3Ry6qmg+5a2/by6+hLxk/n3YKjoI531gPpBjbSxVu0HYonNtF6CGfcj1x5S TN5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721139494; x=1721744294; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=WlfjU1+uJOmDYTuGwaL5xHA70bWDs/47rYtMBUB2API=; b=NMqyR6sMzNxBE1uX7tEuxEvQ5LLjfjlrfsa+s+OP+O5OSPBNdMNkLhIjF1j/lxvXBD rTHHHUmI+KCJJf1DLxfLwjdYhyiX7+36q+6H/5CiClle6XHmD/06p8z/iPxaca6PPegL XcXQRvkkwBAX/p/HBnlsJo4He2RVQDW0DbiIzk78Yrn+CNy1SAZJ21eO9auZeLax/mKR iZW7sNy93RqEmM+0PZow7TsI9TY9NPK+ma33ntDLJLVVF5p2TEr/7GyMjbqVe4Z23lVQ GgI1f+seXGH+2djlh4si32r+dXGaFdrRR9HbY4g7PVySBMCW+bZoggUlPG58NzBXHy+I zQTQ== X-Forwarded-Encrypted: i=2; AJvYcCXOtkmkaUS4YFMwCj0t3qW+B3gasbsRgxa+UJPTuv/2ojV9CthqKB86QqisZt7FmtFumJsS/UbfCPssF2F+QhFlqzA= X-Gm-Message-State: AOJu0Yy+2MONV7twVCHJFH0usExYPRqefADjtc43cMMBiBmaWdO076e1 Fr1PHFJD+yB4QIKhtZVnHoUGa6x4+JNeUQcJVT0DpSeGaTn29HPB X-Google-Smtp-Source: AGHT+IGQjdjJUbhBeyxEgnBt2zTvIYjytX/SASyWiUCoTHSBsvGU0DTYBRHOxMuFj8HGtCKlYSqP0g== X-Received: by 2002:a50:aa93:0:b0:59f:9fc7:1e66 with SMTP id 4fb4d7f45d1cf-59f9fc721bcmr523379a12.9.1721139494100; Tue, 16 Jul 2024 07:18:14 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6402:4310:b0:599:9c73:c392 with SMTP id 4fb4d7f45d1cf-5999c73c484ls2421931a12.2.-pod-prod-04-eu; Tue, 16 Jul 2024 07:18:12 -0700 (PDT) X-Received: by 2002:a05:6402:210b:b0:57c:672b:ca34 with SMTP id 4fb4d7f45d1cf-59ef01be830mr1701555a12.28.1721139491565; Tue, 16 Jul 2024 07:18:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721139491; cv=none; d=google.com; s=arc-20160816; b=ooOFwoJhiLPMsL6IbIMRDku1pClUnZXOMGM9T8x6PcY969J4Ws7+Yzbszcckm3IpoU KA7cHJesZViRCtCkWKTk40TakIvrjAKA1yPd647+EfJtyWbsPQwQPZH3csTmk/NtyN02 QzFv+bFQbxDy51LbR8lOLrNEaXCyMrnuc5CdKhYB/XfC77aIMIxTJX9vouDcKYZPlqNH ACpa8QmnTe+NenQBf8P6m2Pfjme/GO4I2NSIxbZLs5Vo4K1og1l63BuA6b2/Filt5B5U yCMZPP/C/3enjqip4v+recPWjqR3EJD3bmkACXw/Wpc4ueD3Hj2aTAkmP9fQTteeBzxa +p1Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=O6OIIUZOAeri4B2tkgLzjLGvlfizpgiO81DeH8YO/nY=; fh=sG8mVX6bNXJXg1RhSNhryk9YKHryCWWF37H72hfFhEU=; b=ySOzk2+RuPxdXReAVqTjgewCNC6rXJRK0XvmFqzVv69EHLcPBn3AcpPRR1oiMqQGey QUdThfL++z/PzTRTbCZFZuiojUbgF+JO7zE5BXJeuo9Nd6DGx/sSt/31U3AltyvRBbEw GrkC9uOPz2LLWZMlZW5uWsi77m8aER5rY+HLwgEvVDGNkDGBw3oj09J5wseNXdgrtfA9 8aPLfM2IRe9a/jzTZ1CYjAoniKJZWjkAHPFpyAeXIwGA6fV8mbfC1ldG2T2pfTI3jDhi KUibKrgwetqY+u6bfaP5qjz4rGTgoYjt1nTOjd/4G7YPDCJWVcuMXD4IHOsfWx9vQFgZ Iydg==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=BFfPEadz; spf=pass (google.com: domain of fm-294854-202407161418115ce2812f9612c0cc2e-xq_fc2@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-294854-202407161418115ce2812f9612c0cc2e-xq_fc2@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-64-225.siemens.flowmailer.net (mta-64-225.siemens.flowmailer.net. [185.136.64.225]) by gmr-mx.google.com with ESMTPS id 4fb4d7f45d1cf-59b27a0af16si219724a12.5.2024.07.16.07.18.11 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Jul 2024 07:18:11 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-202407161418115ce2812f9612c0cc2e-xq_fc2@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) client-ip=185.136.64.225; Received: by mta-64-225.siemens.flowmailer.net with ESMTPSA id 202407161418115ce2812f9612c0cc2e for ; Tue, 16 Jul 2024 16:18:11 +0200 From: "'Jan Kiszka' via isar-users" To: isar-users Cc: Silvano Cirujano-Cuesta , Benedikt Niedermayr , Felix Moessbauer Subject: [PATCH v3 2/5] container-loader: Introduce helper to load container images into local registry Date: Tue, 16 Jul 2024 16:18:06 +0200 Message-ID: <3b7b8dbdde7fa3a4184daa3f8d567e72c8b50d2d.1721139489.git.jan.kiszka@siemens.com> In-Reply-To: References: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=BFfPEadz; spf=pass (google.com: domain of fm-294854-202407161418115ce2812f9612c0cc2e-xq_fc2@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-294854-202407161418115ce2812f9612c0cc2e-xq_fc2@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Content-Type: text/plain; charset="UTF-8" Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: duEUV0xw8Fp8 From: Jan Kiszka This allows to write dpkg-raw recipes which packages archived container images and load them into a local docker or podman registry on boot. The scenario behind this is to pre-fill local registries in a way that still permits live updates during runtime. The loader script only process images which are not yet available under the same name and tag in the local registry. Also after loading, the archived images stay on the local file system. This allows to perform reloading in case the local registry should be emptied (e.g. reset to factory state). To reduce the space those original images need, they are compressed, by default with zstd. Separate include files are available to cater the main container engines, one for docker and one for podman. Signed-off-by: Jan Kiszka --- .../container-loader/container-loader.inc | 101 ++++++++++++++++++ .../container-loader/docker-loader.inc | 10 ++ .../files/container-loader.service.tmpl | 12 +++ .../files/container-loader.sh.tmpl | 18 ++++ .../container-loader/podman-loader.inc | 10 ++ 5 files changed, 151 insertions(+) create mode 100644 meta/recipes-support/container-loader/container-loader.inc create mode 100644 meta/recipes-support/container-loader/docker-loader.inc create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl create mode 100644 meta/recipes-support/container-loader/podman-loader.inc diff --git a/meta/recipes-support/container-loader/container-loader.inc b/meta/recipes-support/container-loader/container-loader.inc new file mode 100644 index 00000000..e97e829b --- /dev/null +++ b/meta/recipes-support/container-loader/container-loader.inc @@ -0,0 +1,101 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +FILESPATH:append := ":${FILE_DIRNAME}/files" + +inherit dpkg-raw + +SRC_URI += " \ + file://container-loader.service.tmpl \ + file://container-loader.sh.tmpl" + +CONTAINER_COMPRESSION ?= "zst" +CONTAINER_DELETE_AFTER_LOAD ?= "0" + +DEBIAN_DEPENDS += " \ + ${CONTAINER_ENGINE_PACKAGES} \ + ${@', gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \ + ', zstd' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \ + ', xz-utils' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \ + ''}" + +CONTAINER_COMPRESSOR_CMD = "${@ \ + 'gzip -f -9 -n --rsyncable' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \ + 'xz -f ${XZ_DEFAULTS}' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \ + 'zstd -f --rm ${ZSTD_DEFAULTS}' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \ + ''}" + +CONTAINER_DECOMPRESSOR_CMD = "${@ \ + 'gzip -c -d -n' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \ + 'xz -c -d -T0' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \ + 'pzstd -c -d' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \ + ''}" + +TEMPLATE_FILES += " \ + container-loader.service.tmpl \ + container-loader.sh.tmpl" +TEMPLATE_VARS += " \ + CONTAINER_ENGINE \ + CONTAINER_DECOMPRESSOR_CMD \ + CONTAINER_DELETE_AFTER_LOAD" + +do_install() { + install -m 755 ${WORKDIR}/container-loader.sh ${D}/usr/share/${BPN} +} +do_install[cleandirs] += " \ + ${D}/usr/share/${BPN} \ + ${D}/usr/share/${BPN}/images" + +python do_install_fetched_containers() { + import os + + workdir = d.getVar('WORKDIR') + D = d.getVar('D') + BPN = d.getVar('BPN') + + image_list = open(D + "/usr/share/" + BPN + "/image.list", "w") + + src_uri = d.getVar('SRC_URI').split() + for uri in src_uri: + scheme, host, path, _, _, parm = bb.fetch.decodeurl(uri) + if scheme != "docker": + continue + + image_name = host + (path if path != "/" else "") + unpacked_image = workdir + "/" + image_name.replace('/', '.') + dest_dir = D + "/usr/share/" + BPN + "/images" + tar_image = dest_dir + "/" + image_name.replace('/', '.') + ".tar" + docker_ref = ":" + parm["tag"] if "tag" in parm else "latest" + + bb.utils.remove(tar_image) + cmd = f"skopeo copy dir:{unpacked_image} " \ + f"docker-archive:{tar_image}:{image_name}{docker_ref}" + bb.note(f"running: {cmd}") + bb.process.run(cmd) + + cmd = f"{d.getVar('CONTAINER_COMPRESSOR_CMD')} {tar_image}" + bb.note(f"running: {cmd}") + bb.process.run(cmd) + + line = f"{os.path.basename(tar_image)}.{d.getVar('CONTAINER_COMPRESSION')} " + \ + image_name + docker_ref + bb.note(f"adding '{line}' to image.list") + image_list.write(line + "\n") + + image_list.close() +} + +addtask install_fetched_containers after do_install before do_prepare_build + +do_prepare_build:append() { + install -v -m 644 ${WORKDIR}/container-loader.service ${S}/debian/${BPN}.service + + # Do not compress the package, most of its payload is already, and trying + # nevertheless will only cost time without any gain. + cat <> ${S}/debian/rules +override_dh_builddeb: + dh_builddeb -- -Znone +EOF +} diff --git a/meta/recipes-support/container-loader/docker-loader.inc b/meta/recipes-support/container-loader/docker-loader.inc new file mode 100644 index 00000000..b864c854 --- /dev/null +++ b/meta/recipes-support/container-loader/docker-loader.inc @@ -0,0 +1,10 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +require container-loader.inc + +CONTAINER_ENGINE = "docker" + +CONTAINER_ENGINE_PACKAGES ?= "docker.io, apparmor" diff --git a/meta/recipes-support/container-loader/files/container-loader.service.tmpl b/meta/recipes-support/container-loader/files/container-loader.service.tmpl new file mode 100644 index 00000000..1638eaf2 --- /dev/null +++ b/meta/recipes-support/container-loader/files/container-loader.service.tmpl @@ -0,0 +1,12 @@ +[Unit] +Description=Load archived container images on boot +After=${CONTAINER_ENGINE}.service +Requires=${CONTAINER_ENGINE}.service + +[Service] +Type=oneshot +ExecStart=/usr/share/${BPN}/container-loader.sh +RemainAfterExit=true + +[Install] +WantedBy=multi-user.target diff --git a/meta/recipes-support/container-loader/files/container-loader.sh.tmpl b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl new file mode 100755 index 00000000..b6abec92 --- /dev/null +++ b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl @@ -0,0 +1,18 @@ +#!/bin/sh +# +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +set -eu + +while read -r image ref; do + if [ -e /usr/share/${BPN}/images/"$image" ] && \ + [ -z "$(${CONTAINER_ENGINE} images -q "$ref")" ]; then + ${CONTAINER_DECOMPRESSOR_CMD} /usr/share/${BPN}/images/"$image" | \ + ${CONTAINER_ENGINE} load + if [ "${CONTAINER_DELETE_AFTER_LOAD}" = "1" ]; then + rm -f /usr/share/${BPN}/images/"$image" + fi + fi +done < /usr/share/${BPN}/image.list diff --git a/meta/recipes-support/container-loader/podman-loader.inc b/meta/recipes-support/container-loader/podman-loader.inc new file mode 100644 index 00000000..d2c9a12d --- /dev/null +++ b/meta/recipes-support/container-loader/podman-loader.inc @@ -0,0 +1,10 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +require container-loader.inc + +CONTAINER_ENGINE = "podman" + +CONTAINER_ENGINE_PACKAGES ?= "podman" -- 2.43.0 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/3b7b8dbdde7fa3a4184daa3f8d567e72c8b50d2d.1721139489.git.jan.kiszka%40siemens.com.