From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Tue, 09 Jul 2024 19:31:50 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wm1-f56.google.com (mail-wm1-f56.google.com [209.85.128.56]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 469HVn8Y006523 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 9 Jul 2024 19:31:49 +0200 Received: by mail-wm1-f56.google.com with SMTP id 5b1f17b1804b1-426d0bead0asf4850475e9.1 for ; Tue, 09 Jul 2024 10:31:49 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1720546304; cv=pass; d=google.com; s=arc-20160816; b=nxM8ers9RWK3Eu0mWYztGdw8hYZ1VQ74tfdf0GqOJU1f05g9wbXKuuOYvgNKfwptsC 7RvQ5wGN0bPtsMRDTSmfIHl7YZl8vmbHMR50aVjf4OnK+WdECTV1tHTX9SnCeIFAHhw4 fvYPf/qgGM1ECNBJ0TIBctlzxFL1czB7stYvMnxbmaVIydawLT5JlgtuvP5sX6YOzTJp MiLvWHD4f+HSrU31pmxlcEf5NIPTu5WSVaQTpnaufKpLizsfw3SkO4sGK2VZcBMeQ4P/ ofcmDVZtLYas1wgTaTqO3Bzslz/Pp+19kghXI9zIcvzdhoBlcgsePSiCxu25Sj8ArkFX N+sA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=9KFqqe0+zUNuf2b3S+LuEbkWCkj4tHfVFrRH80sjhzs=; fh=KMhMelnZUL30TyZPHbQHWxTh5xbYk8N1phqfO3lJ90w=; b=MSuQOUnn44nx1iOfiABw1m7SBVl18Pyv0+aoPUwzEToDXPNmENpIYUbYQjKlTMs0p8 2+OAAzkNJ7Rn7FSXSrNOVWDyBJl0jppfgKE/bTYBauU6V45DShCWyVzk3C94XnsDV8Jg vIhr2XZbThcK+jWTQ2ubGDiICXpbssB8fvKoQcPg4Eop73WQsintI5bT2i6p3wbgQgyq IFW0wTjiyo0ww1EsngkHh+6qo4Miy7DMXANQLMDmsg+vp6j57RND0BRaSNw0dw53ZzNO tPe3wnBaAcBPkUChYXSJeNAaQ9Nt/EMVeyEIsZhWf+CQHi3Ux3Rse/9V63pHckGkGUGB VzBA==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=BxUVJ49q; spf=pass (google.com: domain of fm-294854-20240709173140ddaf302001721ef599-8lvdhi@rts-flowmailer.siemens.com designates 185.136.64.228 as permitted sender) smtp.mailfrom=fm-294854-20240709173140ddaf302001721ef599-8LvdHi@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1720546304; x=1721151104; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:to:from :from:to:cc:subject:date:message-id:reply-to; bh=9KFqqe0+zUNuf2b3S+LuEbkWCkj4tHfVFrRH80sjhzs=; b=tVRl5DKuA1wS+VYDmaC8SI0e2OUgDt8rLnlCGfz/GKlggq6K778nyPbRA565n9mXDf UsNfhdkDZj6l8vy0ujCwDI80urvULZ0kYtdGx/SORzOSsupqD++glc8C+eiKgxoSjkG+ ITtD7NxTIgoIJr/twmbPrjUNMDZmOQ9hxL2f/MFPvabLCNxCV0+ywXh7a4gNw15VzjQt VkqRnV+mObDNSNSDPgcY65JYuwYJKVqrL6cb55WAVEjYLUD/FNbIcockSIjUR4kIFtER Vz8HcVTaln1Y8HTQTXUS6dvZsPhfBAGGxY2o0BRF4F4PwymUXPuAglMOCUH/IrmEx9/9 QVLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720546304; x=1721151104; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:to:from :x-beenthere:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=9KFqqe0+zUNuf2b3S+LuEbkWCkj4tHfVFrRH80sjhzs=; b=kYCKRFJolYd6nGqUF3Mm7EZQOYFdNeJQej+8pq46eJEv+4LhNVXmy/DxcvRn6YnPce 30PQGft8F73qgV9JqsJDlJYx/gRGyjRZWoGaGE258elWz08/Fk0Lwt2fQAPk1/u2S6Y7 udDlwtFL8qhujvBlj+70s0i1dIbDhnkRZQjqaPfUjRMLrCmHMntkPu9Nnul46CG6DtM0 w6Sqwr6tVmIdafiqk2hW/DYZewfYmYaZcgMm6bnw7kkuGWyhPqnEpcFTScHixnDl4tmI GX/YJpkcPdNs9+4lq303WVZ58kO8KeML9plsm22vkOSkaGZv4wgta9dRuAJmJtPGT8KA Am5w== X-Forwarded-Encrypted: i=2; AJvYcCUh58+9a8gIngodNCm55+CHvt6XbebRCSUSAdwPPFtyfaxSwtxh1vuHPpkuLhh1cL1KzgslNeRexYHM7KaOZ4u5LG8= X-Gm-Message-State: AOJu0YwAZstfyO7F+E+/jjOuiy7zRu8cR4P2ZAc89zQ2oqmB9FbH3qkV J/vTweDWJr0nr5zbPVOiTx5ViFywr1W8jJB3hdVAM+ogJOGIhhJe X-Google-Smtp-Source: AGHT+IEqrfNvHf47zObvKMmE00TDrqxXlXjjSEudiBz2D49RPGGE50DE4+Kp8e1PgImprDSE9/ah3g== X-Received: by 2002:a05:600c:4ba2:b0:426:6ead:5709 with SMTP id 5b1f17b1804b1-426707cf736mr19336035e9.9.1720546303397; Tue, 09 Jul 2024 10:31:43 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:1d0f:b0:427:9377:3cc4 with SMTP id 5b1f17b1804b1-42793774063ls94135e9.1.-pod-prod-06-eu; Tue, 09 Jul 2024 10:31:41 -0700 (PDT) X-Received: by 2002:a05:600c:4341:b0:424:ad14:6b79 with SMTP id 5b1f17b1804b1-426706c9159mr23700795e9.8.1720546301252; Tue, 09 Jul 2024 10:31:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1720546301; cv=none; d=google.com; s=arc-20160816; b=X3kRsZe08+yj2m3sBtOSLuq6/h4kMGaXEW4p/DnJ2P+ydDuPLyh+RFieEeh9WoJLx9 o/IfGEIuCx1YI76XC/l48p0pOd2DYe3RD0hVCZBzsZ1jvXK/7ruMS/hYS7NA/6XMfhxT q9zqsZG1Ps4HZT3uzuWw3A0gLNnwoWoHhHyRsoODOiqZnVlBz0WJOwhFnKwX683sF+/D 52cQzr/Sp4hSEuYfU/MrCwiomHIO+KesC7Xh73cBsglbnTjVXOy7jBcI4DkjyN02B5C8 6zdMvH67urCUCCvtQ1JzTbg2Ee6V7Po4BjgUHJ5f6OeKhoc3aM6WV/k2OX30rlNWcfW7 ccWQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:to:from:dkim-signature; bh=L8JXHy4gkDOvNTnuYGp+qJaSa8/+nONEOLnUEdxdg5k=; fh=hOBXq6nisExkIRIwN8IDOmOCLKwX8PM0ANlxQbNMEqA=; b=E6KNMnoCk5v+UzF2/oao2KvYbQkKCxy4wuuBGMT/55O2fnfOvMLsxdv09ZtfktZMsz sj2o1/KYykRlkz5TFhF/I4gWfhKIfBMxLHmBrFUEmLptKXgjdbthcEJugNqjZPQ6TIRN d7SyrNoCZwApcQ5ntQ8bVLBw682O/h87cK6cfCMHUKLj3XS7RYj6oF5RSyeAfJZ92vPo ZfpO8FZrheewZ+9j/wc0pa3wHZ/Zbt4W+GPXv3Tk4qA6UrANKNo2BEc1rc72btK6sYaj /GcXkhYpRMjXisw86pOhzETs+WRJ2Mbr13sxZklo8gf4zxtUtiu6i/1a+4sjOAgzNNk9 53iw==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=BxUVJ49q; spf=pass (google.com: domain of fm-294854-20240709173140ddaf302001721ef599-8lvdhi@rts-flowmailer.siemens.com designates 185.136.64.228 as permitted sender) smtp.mailfrom=fm-294854-20240709173140ddaf302001721ef599-8LvdHi@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-64-228.siemens.flowmailer.net (mta-64-228.siemens.flowmailer.net. [185.136.64.228]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-426725584d2si626425e9.0.2024.07.09.10.31.41 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Jul 2024 10:31:41 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-20240709173140ddaf302001721ef599-8lvdhi@rts-flowmailer.siemens.com designates 185.136.64.228 as permitted sender) client-ip=185.136.64.228; Received: by mta-64-228.siemens.flowmailer.net with ESMTPSA id 20240709173140ddaf302001721ef599 for ; Tue, 09 Jul 2024 19:31:40 +0200 From: "'Jan Kiszka' via isar-users" To: isar-users Subject: [RFC][PATCH 2/3] container-loader: Introduce helper to load container images into local registry Date: Tue, 9 Jul 2024 19:31:38 +0200 Message-ID: In-Reply-To: References: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=BxUVJ49q; spf=pass (google.com: domain of fm-294854-20240709173140ddaf302001721ef599-8lvdhi@rts-flowmailer.siemens.com designates 185.136.64.228 as permitted sender) smtp.mailfrom=fm-294854-20240709173140ddaf302001721ef599-8LvdHi@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Content-Type: text/plain; charset="UTF-8" Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: IEtz6r3UPSm8 From: Jan Kiszka This allows to write dpkg-raw recipes which packages archived container images and load them into a local docker or podman registry on boot. The scenario behind this is to pre-fill local registries in a way that still permits live updates during runtime. The loader script only process images which are not yet available under the same name and tag in the local registry. Also after loading, the archived images stay on the local file system. This allows to perform reloading in case the local registry should be emptied (e.g. reset to factory state). To reduce the space those original images need, they are compressed, by default with xz. Separate include files are available to cater the main container engines, one for docker and one for podman. Signed-off-by: Jan Kiszka --- .../container-loader/container-loader.inc | 76 +++++++++++++++++++ .../container-loader/docker-loader.inc | 10 +++ .../files/container-loader.service.tmpl | 11 +++ .../files/container-loader.sh.tmpl | 13 ++++ .../container-loader/podman-loader.inc | 10 +++ 5 files changed, 120 insertions(+) create mode 100644 meta/recipes-support/container-loader/container-loader.inc create mode 100644 meta/recipes-support/container-loader/docker-loader.inc create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl create mode 100644 meta/recipes-support/container-loader/podman-loader.inc diff --git a/meta/recipes-support/container-loader/container-loader.inc b/meta/recipes-support/container-loader/container-loader.inc new file mode 100644 index 00000000..8e352214 --- /dev/null +++ b/meta/recipes-support/container-loader/container-loader.inc @@ -0,0 +1,76 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +FILESPATH:append := ":${FILE_DIRNAME}/files" + +inherit dpkg-raw + +SRC_URI += " \ + file://container-loader.service.tmpl \ + file://container-loader.sh.tmpl" + +TEMPLATE_FILES += " \ + container-loader.service.tmpl \ + container-loader.sh.tmpl" +TEMPLATE_VARS += "CONTAINER_ENGINE" + +CONTAINER_COMPRESSION ?= "xz" + +DEBIAN_DEPENDS += " \ + ${CONTAINER_ENGINE_PACKAGES} \ + ${@', xz-utils' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \ + ', gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \ + ''}" + +CONTAINER_COMPRESSOR = "${@ \ + 'xz' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \ + 'gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \ + ''}" + +python do_install() { + import os + + workdir = d.getVar('WORKDIR') + D = d.getVar('D') + PN= d.getVar('PN') + + image_list = open(D + "/usr/share/" + PN +"/image.list", "w") + + src_uri = d.getVar('SRC_URI').split() + for uri in src_uri: + scheme, host, path, _, _, parm = bb.fetch.decodeurl(uri) + if scheme != "docker": + continue + + image_name = host + (path if path != "/" else "") + unpacked_image = workdir + "/" + image_name.replace('/', '.') + dest_dir = D + "/usr/share/" + PN + "/images" + tar_image = dest_dir + "/" + image_name.replace('/', '.') + ".tar" + docker_ref = ":" + parm["tag"] if "tag" in parm else "latest" + + cmd = f"skopeo copy dir:{unpacked_image} " \ + f"docker-archive:{tar_image}:{image_name}{docker_ref}" + bb.note(f"running: {cmd}") + bb.process.run(cmd) + + cmd = f"{d.getVar('CONTAINER_COMPRESSOR')} {tar_image}" + bb.note(f"running: {cmd}") + bb.process.run(cmd) + + line = f"{os.path.basename(tar_image)}.{d.getVar('CONTAINER_COMPRESSION')} " + \ + image_name + docker_ref + bb.note(f"adding '{line}' to image.list") + image_list.write(line + "\n") + + image_list.close() + + bb.utils.copyfile(workdir + "/container-loader.sh", + D + "/usr/share/" + PN + "/container-loader.sh") +} +do_install[cleandirs] += "${D}/usr/share/${PN}/images" + +do_prepare_build:append() { + install -v -m 644 ${WORKDIR}/container-loader.service ${S}/debian/${PN}.service +} diff --git a/meta/recipes-support/container-loader/docker-loader.inc b/meta/recipes-support/container-loader/docker-loader.inc new file mode 100644 index 00000000..b864c854 --- /dev/null +++ b/meta/recipes-support/container-loader/docker-loader.inc @@ -0,0 +1,10 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +require container-loader.inc + +CONTAINER_ENGINE = "docker" + +CONTAINER_ENGINE_PACKAGES ?= "docker.io, apparmor" diff --git a/meta/recipes-support/container-loader/files/container-loader.service.tmpl b/meta/recipes-support/container-loader/files/container-loader.service.tmpl new file mode 100644 index 00000000..afde55d3 --- /dev/null +++ b/meta/recipes-support/container-loader/files/container-loader.service.tmpl @@ -0,0 +1,11 @@ +[Unit] +Description=Load archived container images on boot +After=${CONTAINER_ENGINE}.service + +[Service] +Type=oneshot +ExecStart=/usr/share/${PN}/container-loader.sh +RemainAfterExit=true + +[Install] +WantedBy=multi-user.target diff --git a/meta/recipes-support/container-loader/files/container-loader.sh.tmpl b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl new file mode 100755 index 00000000..31d27865 --- /dev/null +++ b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl @@ -0,0 +1,13 @@ +#!/bin/sh +# +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +set -eu + +while read -r image ref; do + if [ -z "$(${CONTAINER_ENGINE} images -q "$ref")" ]; then + ${CONTAINER_ENGINE} load -i /usr/share/${PN}/images/"$image" + fi +done < /usr/share/${PN}/image.list diff --git a/meta/recipes-support/container-loader/podman-loader.inc b/meta/recipes-support/container-loader/podman-loader.inc new file mode 100644 index 00000000..d2c9a12d --- /dev/null +++ b/meta/recipes-support/container-loader/podman-loader.inc @@ -0,0 +1,10 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +require container-loader.inc + +CONTAINER_ENGINE = "podman" + +CONTAINER_ENGINE_PACKAGES ?= "podman" -- 2.43.0 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/ba72fae1b0dff34ff4474a2cd53939a6c4fd3279.1720546299.git.jan.kiszka%40siemens.com.