From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 19 Jul 2024 18:38:57 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f190.google.com (mail-lj1-f190.google.com [209.85.208.190]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46JGcsNV002655 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 19 Jul 2024 18:38:54 +0200 Received: by mail-lj1-f190.google.com with SMTP id 38308e7fff4ca-2ee94b0e2e1sf27833551fa.2 for ; Fri, 19 Jul 2024 09:38:54 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721407127; cv=pass; d=google.com; s=arc-20160816; b=qKhyGQ2L8UaG1jpZzAcCWTAod5Go32fJX6nwtbOw7PXx2ia0txnmJWO1cqq7Uetd4Y qIOQo+L8Fm8ItnW9AfzUFRVBkHnNsCZDMg3/cTq0rxVmcYV36UqicOt9vOLcUTOwDY82 UOLogBxhJpqXk/AHM1v+HZFAzhNCVjfLpXaruNVm0MOQ7a2xR0V76kLZPsB9fVhy2j6/ wh3h5mfGLBEHXue98zT2+GKlnS6WK4j3To/ZNDdY+Af377OigVuOIfphtisFyxAeBWDU VxFl65ha0s8lDM0nO7AF9mq7BXOPZ5okv50jPTNpyqVCAhIA7WYOTFxL39cxp6pazqjM wBaw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ixR0Rm/MjDuYNur1OagzZfinUR3M9IjpWNAhho4ZejI=; fh=YpxM6NPb3ktQymkigQLsvyZHDlkZm13v91eNiiFISpc=; b=xjtdv1orKOMJAWaG8o3f3oQmrrmuYXKjGLUVHnf+csWS/LuMhkCmC2PPH1Ut1w2mvI xGG+B7JeRMWConp6croPLuFG0IvM2r9qLkxsak5baifLRqZDRaF6/MebyEn47ipLmfJ0 a1c9qvOGXDQ2piGuxnQ2zTfSB2NNAEy88YVukow9C+/yvLsdc9pOYBSM4/gugaPfq7m6 ob+JuECU8CHXG5TMef8CG3fGYfqCWdHWsfCSSglLAfeal/0OhLB8HWA3fWcGO4FrI0tB FxZzV1qvhawUo2MgcOYzHmB6hxU05MHENQ8lGwdjc8E3HdF5l2a5KHVg9IfibOh1WAOZ 2CFQ==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=GgfuQWmH; spf=pass (google.com: domain of fm-294854-20240719163844b8f4655d42b9dc012d-zyscph@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-294854-20240719163844b8f4655d42b9dc012d-zYsCpH@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721407127; x=1722011927; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=ixR0Rm/MjDuYNur1OagzZfinUR3M9IjpWNAhho4ZejI=; b=hefXgHDDK5wv+BHN3aUGj1Y9On63AsCSABG0JfluIFprXPyXOrRTxjm6IcaQJiWil/ 6SYg085s+4TME5Z1cBvdx+BNLn83ZVOBm/0KGwNko4BMBQgZy6V2ePaITbSMQT6csVi+ C8NgpN+fzUZkGIVmKcTNO1bbUKMw9Aw4kM2Pcthn2O59MYx1fmxrZpxg7q2CVrGkdcNj hUR54yg7FX68OBRQey71DxAKiC8NRl4f44jBSajKkiUG6FbnTgTJYnPs/Pcm2uC4k6cq usg2C5tht46/DtCkeaU/mzsGwA3sYhtlqwK+MezhmHvAXsDHhz0erghQZMBkBBaDLaIr RS1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721407127; x=1722011927; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=ixR0Rm/MjDuYNur1OagzZfinUR3M9IjpWNAhho4ZejI=; b=Aqjp0BR5lNvFjs9wUQPtjWW89h39g7Ijly4NWH1pzERU82j3BcInQLGytzZvoryrvf ltXKUI1XoJyZIlluDDf8LE3v27T6jHOjFOgExn8YPrBNQ8+FVo9n9FLxStP/EfEO80pm yyaUzuR4rhWyGHkaZsoZ2yktannXl667FA80QOYy89sZ62/pDCvY7GSPxvy42sdxKlxx u5xPDXu6eVidw1+fpBnbLoVGml2zRsE7moTDWPIPRzgQj9aHdJ9c2QC5wciiDeHBGl4m Y9lInELtmNWI2DvJAOzEgqwZVkBToeQwLO4OR9wGAl9jh63O0hMjS331w3hon6/RobwY a95w== X-Forwarded-Encrypted: i=2; AJvYcCW4zgLRLOKBKWxR1GPtPTfn/vMSW1OylUMdRBb4EmhBcalGw7WhLb+94j10St00sDrHkyWNieS5Ic3gf3qwGB+3Yg0= X-Gm-Message-State: AOJu0YxNl4j1iDV/zxC1znTCuMj2+ngl4ev/QXo1ESZ+Yb14SpmkH88m bt4h8ZkFFmWazd8UNLRht/omS1WT7SrhOV5msybypZTPzpTrfzoT X-Google-Smtp-Source: AGHT+IGuNM6BLXiKYzgLddi66zjBoQN/AKFBVlqMqSXh4BoIcQSG03vKSyb01CqwQCbMBkQKmNfOvA== X-Received: by 2002:a2e:a404:0:b0:2ee:8c8d:d9dd with SMTP id 38308e7fff4ca-2ef16840f94mr2137721fa.36.1721407127003; Fri, 19 Jul 2024 09:38:47 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:a9a4:0:b0:2ee:605b:3d02 with SMTP id 38308e7fff4ca-2ef04dda89cls12094081fa.2.-pod-prod-01-eu; Fri, 19 Jul 2024 09:38:45 -0700 (PDT) X-Received: by 2002:a2e:9101:0:b0:2ec:440c:4e1c with SMTP id 38308e7fff4ca-2ef167824f8mr2177381fa.11.1721407124619; Fri, 19 Jul 2024 09:38:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721407124; cv=none; d=google.com; s=arc-20160816; b=HFDMVULEFGPvIHdxCPEcDfqh3wlsa0ws1/6O5+k9Vj+r+jLZRC33QeSDrwJ2o53Jvp XJ8tuPKI2v1CDyHgpEcFo2YoFEsJG5IivYIn1KSicPHl2nkHAgRVyI5jWDn0ZHRmKAEN QhvWkHyJPk8REogjJnpvJeCM6Sw5XPvfq7pkgC+FkpDiIHnES3naaEDHa1pw0lGJZMkY qK6vE/zHHz5f4a0urOUhNdwwA82fp/Cek4yP/qJ3tA6OEZSSCTjv+xCoj+QO768ES5kX b3SmGsOJLtNMwWSA3cu2Psu2k6GHGQWv+rU9PDS5zY5rPXvaKD27V/xWQ9CdppZxLilC OORg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=YTKlvg2yRZslJX8lHMaY1WIwEOdoVww+2lzK9eEeB0c=; fh=sG8mVX6bNXJXg1RhSNhryk9YKHryCWWF37H72hfFhEU=; b=JoYzezORfijh41zLWwr86FL+yCPEYGzs2/ELKXGOsIYMmpS8o5H+ul5S9axDXRFdP8 xuBykVBXjMycM/sJ9B7NSQfoA794kNbf1oE5I86x7FkecqQgkKpz/T7V9X6kObhq1EzM VZmX6MCQjcG+tce2w9f9qX9LY97SHe2vm+/cO1uNDUuODeqpj8x6embN+jIXtzRKEfEq kSbFcl6hy8UYr7MsBpIaqgm62FaqtZvzfwR0Z0PdF9Ik33/6INCsbnGYMkiH9upMIqVJ AwO6VkP6aENlxUQCnsbL/7I8o+0XkxpvMWr7U5f4BkSt4laIVhWw1MWmY9rX+oK5SxfF LLiQ==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=GgfuQWmH; spf=pass (google.com: domain of fm-294854-20240719163844b8f4655d42b9dc012d-zyscph@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-294854-20240719163844b8f4655d42b9dc012d-zYsCpH@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-65-227.siemens.flowmailer.net (mta-65-227.siemens.flowmailer.net. [185.136.65.227]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-427d2911f57si1635545e9.0.2024.07.19.09.38.44 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 19 Jul 2024 09:38:44 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-20240719163844b8f4655d42b9dc012d-zyscph@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) client-ip=185.136.65.227; Received: by mta-65-227.siemens.flowmailer.net with ESMTPSA id 20240719163844b8f4655d42b9dc012d for ; Fri, 19 Jul 2024 18:38:44 +0200 From: "'Jan Kiszka' via isar-users" To: isar-users Cc: Silvano Cirujano-Cuesta , Benedikt Niedermayr , Felix Moessbauer Subject: [PATCH v4 2/5] container-loader: Introduce helper to load container images into local registry Date: Fri, 19 Jul 2024 18:38:40 +0200 Message-ID: <5af6163750f7ae0cb186e52727afe3ced1db2ce2.1721407122.git.jan.kiszka@siemens.com> In-Reply-To: References: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=GgfuQWmH; spf=pass (google.com: domain of fm-294854-20240719163844b8f4655d42b9dc012d-zyscph@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-294854-20240719163844b8f4655d42b9dc012d-zYsCpH@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Content-Type: text/plain; charset="UTF-8" Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: VwrqsdiYVISI From: Jan Kiszka This allows to write dpkg-raw recipes which packages archived container images and load them into a local docker or podman registry on boot. The scenario behind this is to pre-fill local registries in a way that still permits live updates during runtime. The loader script only process images which are not yet available under the same name and tag in the local registry. Also after loading, the archived images stay on the local file system. This allows to perform reloading in case the local registry should be emptied (e.g. reset to factory state). To reduce the space those original images need, they are compressed with zstd. Separate include files are available to cater the main container engines, one for docker and one for podman. Signed-off-by: Jan Kiszka --- .../container-loader/container-loader.inc | 73 +++++++++++++++++++ .../container-loader/docker-loader.inc | 10 +++ .../files/container-loader.service.tmpl | 12 +++ .../files/container-loader.sh.tmpl | 18 +++++ .../container-loader/podman-loader.inc | 10 +++ 5 files changed, 123 insertions(+) create mode 100644 meta/recipes-support/container-loader/container-loader.inc create mode 100644 meta/recipes-support/container-loader/docker-loader.inc create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl create mode 100644 meta/recipes-support/container-loader/podman-loader.inc diff --git a/meta/recipes-support/container-loader/container-loader.inc b/meta/recipes-support/container-loader/container-loader.inc new file mode 100644 index 00000000..5fd8d23c --- /dev/null +++ b/meta/recipes-support/container-loader/container-loader.inc @@ -0,0 +1,73 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +FILESPATH:append := ":${FILE_DIRNAME}/files" + +inherit dpkg-raw + +SRC_URI += " \ + file://container-loader.service.tmpl \ + file://container-loader.sh.tmpl" + +CONTAINER_DELETE_AFTER_LOAD ?= "0" + +DEBIAN_DEPENDS += "${CONTAINER_ENGINE_PACKAGES}, zstd" + +TEMPLATE_FILES += " \ + container-loader.service.tmpl \ + container-loader.sh.tmpl" +TEMPLATE_VARS += " \ + CONTAINER_ENGINE \ + CONTAINER_DELETE_AFTER_LOAD" + +do_install() { + install -m 755 ${WORKDIR}/container-loader.sh ${D}/usr/share/${BPN} +} +do_install[cleandirs] += " \ + ${D}/usr/share/${BPN} \ + ${D}/usr/share/${BPN}/images" + +python do_install_fetched_containers() { + from oe.path import copyhardlink + + workdir = d.getVar('WORKDIR') + D = d.getVar('D') + BPN = d.getVar('BPN') + + image_list = open(D + "/usr/share/" + BPN + "/image.list", "w") + + src_uri = d.getVar('SRC_URI').split() + for uri in src_uri: + scheme, host, path, _, _, parm = bb.fetch.decodeurl(uri) + if scheme != "docker": + continue + + tag = parm["tag"] if "tag" in parm else "latest" + image_name = host + (path if path != "/" else "") + image_file = image_name.replace('/', '.') + \ + ":" + tag + ".zst" + dest_dir = D + "/usr/share/" + BPN + "/images" + + copyhardlink(workdir + "/" + image_file, dest_dir + "/" + image_file) + + line = f"{image_file} {image_name}:{tag}" + bb.note(f"adding '{line}' to image.list") + image_list.write(line + "\n") + + image_list.close() +} + +addtask install_fetched_containers after do_install before do_prepare_build + +do_prepare_build:append() { + install -v -m 644 ${WORKDIR}/container-loader.service ${S}/debian/${BPN}.service + + # Do not compress the package, most of its payload is already, and trying + # nevertheless will only cost time without any gain. + cat <> ${S}/debian/rules +override_dh_builddeb: + dh_builddeb -- -Znone +EOF +} diff --git a/meta/recipes-support/container-loader/docker-loader.inc b/meta/recipes-support/container-loader/docker-loader.inc new file mode 100644 index 00000000..b864c854 --- /dev/null +++ b/meta/recipes-support/container-loader/docker-loader.inc @@ -0,0 +1,10 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +require container-loader.inc + +CONTAINER_ENGINE = "docker" + +CONTAINER_ENGINE_PACKAGES ?= "docker.io, apparmor" diff --git a/meta/recipes-support/container-loader/files/container-loader.service.tmpl b/meta/recipes-support/container-loader/files/container-loader.service.tmpl new file mode 100644 index 00000000..1638eaf2 --- /dev/null +++ b/meta/recipes-support/container-loader/files/container-loader.service.tmpl @@ -0,0 +1,12 @@ +[Unit] +Description=Load archived container images on boot +After=${CONTAINER_ENGINE}.service +Requires=${CONTAINER_ENGINE}.service + +[Service] +Type=oneshot +ExecStart=/usr/share/${BPN}/container-loader.sh +RemainAfterExit=true + +[Install] +WantedBy=multi-user.target diff --git a/meta/recipes-support/container-loader/files/container-loader.sh.tmpl b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl new file mode 100755 index 00000000..2356e31c --- /dev/null +++ b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl @@ -0,0 +1,18 @@ +#!/bin/sh +# +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +set -eu + +while read -r image ref; do + if [ -e /usr/share/${BPN}/images/"$image" ] && \ + [ -z "$(${CONTAINER_ENGINE} images -q "$ref")" ]; then + pzstd -c -d /usr/share/${BPN}/images/"$image" | \ + ${CONTAINER_ENGINE} load + if [ "${CONTAINER_DELETE_AFTER_LOAD}" = "1" ]; then + rm -f /usr/share/${BPN}/images/"$image" + fi + fi +done < /usr/share/${BPN}/image.list diff --git a/meta/recipes-support/container-loader/podman-loader.inc b/meta/recipes-support/container-loader/podman-loader.inc new file mode 100644 index 00000000..d2c9a12d --- /dev/null +++ b/meta/recipes-support/container-loader/podman-loader.inc @@ -0,0 +1,10 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +require container-loader.inc + +CONTAINER_ENGINE = "podman" + +CONTAINER_ENGINE_PACKAGES ?= "podman" -- 2.43.0 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/5af6163750f7ae0cb186e52727afe3ced1db2ce2.1721407122.git.jan.kiszka%40siemens.com.