From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 19 Jul 2024 18:38:56 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f63.google.com (mail-lf1-f63.google.com [209.85.167.63]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46JGcrGw002646 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 19 Jul 2024 18:38:54 +0200 Received: by mail-lf1-f63.google.com with SMTP id 2adb3069b0e04-52e982ad660sf1653881e87.0 for ; Fri, 19 Jul 2024 09:38:54 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721407128; cv=pass; d=google.com; s=arc-20160816; b=benWLilQ0XJfX6e2UrucxM9cSCvISEcEB6+BUSIZaoCcEuJxOW9h2yh23uUmM9li4y q9kpOcO0DZ65xTjK1k5XeFqplJnBMDHSHDHJQ9uqth69kkDJuYOtuLBqIQjoXZyIQiSc bj+E/b6zJf1/b4wEWSrq4CB7NZmCqlc+SWgsfnSPa1N60tOfCBL55ZnpBj9QFoEpPYSA zroV1obp+4LReh5Twl/YjhKu3l94lcHo0eKWYhikOrjgl5v/gp2qXYaYNu3DJm8+xdwE 11o2Id+aWc2sLqufYibfmcAVIvXibiTEQeMW7tucWK/Vpkk/KeLWVA8hx9JT39Jvc3dd OJ9Q== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=PIHes6YNtpIbdr3Sgh6biABdNvVPqlmNtNcWqv9dWzc=; fh=e46DLIVr+6TEdMmHOnfdg0bLxk0m6WQguZ6exyLxRN8=; b=UdroglKjMKH1j+zuzBeLHWuP4O8EVz19XsnuCNR7e7gjnTiYVRC1s4ho8WROqdmrS6 iGQlNYOV2SLH/NgKw54fXK763qShkkd9L6xd34jqPymS6ZADzf6rrcW/Wu1g73c2003T y9mmpmC6OiHJtosfWDJqIx5rMOqZBZQGbnh907/681MhBFMLDS8SV5Vp2pXHZ09cCk/U PbzlVeDhGq1GGwkvdR7h3e+//Q47SvbYA7RJ7YEpGAdu1t8uHjHbXhREzFNsQhP+TOGo XP93Rh66pIXVR94qdfE7ViUeuq7TOhpn7+S3OUm0jrNiLA+jyT4uTmN3AC14lqBlbtPw At6A==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=jbgFU5+C; spf=pass (google.com: domain of fm-294854-20240719163844e28a1b2b4382c7a18d-yjhxtv@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-294854-20240719163844e28a1b2b4382c7a18d-YjHXtv@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721407128; x=1722011928; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=PIHes6YNtpIbdr3Sgh6biABdNvVPqlmNtNcWqv9dWzc=; b=fy28tJ3f0hmYLD8y0h/6+dCdAOgsBwVRiHuk/kkkoj3JtWXD7RQKj39uzjcufLuhd/ 8C+QD+KgdSLy/L86WTiqJOiewfaGvJqhEJbeNcA8ANflBRtkQYakCyub7YvtdaiQG5r/ OaHn+U8752HdzspwLf/xkwWpZjyqx3sQjx5XspKJ5PQdzppy9qCDplpFfpkC1UF7ZX1x bKpNYaROwSHMHTkRl1Nq+QFSVMwwpcbA8ApDMhthH76iY8Q2wxugHFDkE15Fb6ewNH5v s/KJORq3A2L4cFDaghZnAGfgGRRV7ahvvJ7Euw9m9XFMrW6eiW0t0jX4UsJcVxD3KsaY Ql4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721407128; x=1722011928; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=PIHes6YNtpIbdr3Sgh6biABdNvVPqlmNtNcWqv9dWzc=; b=RxFSK/+YAaXBG0AZ9s6l+Upc1qDgL01PLVbDwRn3sKQXn28wbVyT50S30FhhyxLw8B 0hjNCtvpfSkcIXindYEhwI8eEbM1X4doEGBd9Iikm6SY0LVlGuschmCAcJYyzelWXy+G Dp5Mq/AAJo/NYdO+uESKiYfy+QiEt7q50gIz8GPWw2G7D1P6XyjT2FwhhX9OXE6pjM/Z 5/bRIZ9iR6bYMrxmbJJaoWxWStRIOrDes3G5QZoXQtTByGd1ahR9ELGRODFaad79Sj43 dciu0/lVt4buSzvR/d4HaD5ak4f8dNPHhrLF2l4FvLWrEvKPjs/FoKaG5efI2a0i9XsH i3NQ== X-Forwarded-Encrypted: i=2; AJvYcCVnAj/6TGzEavBSrXtajt03hi6PG0qgmKNBW5nBMrXmcqBHBAwAD8j888TElGo/TZOA4ijPubrao8ZsSxuBenpqEMU= X-Gm-Message-State: AOJu0YxQS5nZy1Vsih+FYiDYcssb0Biqj+0XP34x2qF/IdHrmXULZX8d oXZMXmZZuDSj1T+BrR9jGFuDeKOB7OpFJlcKQx+Ys6Pp4LS/1l0f X-Google-Smtp-Source: AGHT+IHaC2gceeeJQruDY5QEsjbXZig7LE2ETEMxod6+WZ1HCBdh1KwzWImRgSvASowScyo9GBPaFQ== X-Received: by 2002:a05:6512:3f13:b0:52c:dc57:868b with SMTP id 2adb3069b0e04-52eeaeb5d8amr4001575e87.13.1721407127305; Fri, 19 Jul 2024 09:38:47 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6512:b0b:b0:52d:259b:377c with SMTP id 2adb3069b0e04-52eeacb4a2fls874116e87.1.-pod-prod-07-eu; Fri, 19 Jul 2024 09:38:45 -0700 (PDT) X-Received: by 2002:a2e:9893:0:b0:2ee:bd1b:84c with SMTP id 38308e7fff4ca-2ef166214f5mr2378601fa.0.1721407124979; Fri, 19 Jul 2024 09:38:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721407124; cv=none; d=google.com; s=arc-20160816; b=pamy7Fi4ghv/6AVkTTFSEYS1ALfuN/z09ikMhW7U3SRiTNB2O2S/w+zjsKSj2CCZ7k SGmWQWcGopxelCvI5rH69L+5z1+xc7TgWem1mV/4Acd5WOE+QG3AwJTAm6Fe/ZfgwBSo cbTtpf49xZSb+08on7Mk3gV1kzsWh/+LH5Uz2ysdcI3E6hiY9KPh4Vf+TVDm2qPhhFp1 vYNH8jn4Dv1FUnPHcGEtRKweNWgRgI73iy5S7nZwLnoGSOZg0GdaHIqT3Hpa6uUEFZal PG1c1Uvu2b3Fg2COFES1pxFe0WVY910iXSUa88mchSWzsiT1epXYOXr/27ZA6b8DmrMw E/8w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=tur0DYltz4BuUK9fghA1yjJMijPlyfHlWAkmvVWKtn4=; fh=sG8mVX6bNXJXg1RhSNhryk9YKHryCWWF37H72hfFhEU=; b=YR1f/i+xrRysNZxKzcftpZT6n1rsjBJJV9uuOTcUqL65nf4msVH82P4ODZKeoyuYzO l0Ya/99XleAQ8GGzNSOmQPqonIUTSoujk4A8s8GpJ2ypPZvBmXJ8cH3y0946nCrdi0G8 1R/DcQegGDsosB4FdUDUtxdLZLouduLCdfIfWZYj0jjx2vXSbra2goP/AOY9gLxDgdJB ZoffPBbiFwHNLSuxCHo7Ylu0TSOtTMKmM+ZeqwHLH6NUiHNGSQe3GNm21GLIDdhCsz4x bTA0ZcHUIuInbs+7SkqQxXazaBSXqW19/d7weGgqWrc3BGANKhekwzexC9u0kvwl5wG8 bZyg==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=jbgFU5+C; spf=pass (google.com: domain of fm-294854-20240719163844e28a1b2b4382c7a18d-yjhxtv@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-294854-20240719163844e28a1b2b4382c7a18d-YjHXtv@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-64-225.siemens.flowmailer.net (mta-64-225.siemens.flowmailer.net. [185.136.64.225]) by gmr-mx.google.com with ESMTPS id 38308e7fff4ca-2ef0fd30618si385501fa.8.2024.07.19.09.38.44 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 19 Jul 2024 09:38:44 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-20240719163844e28a1b2b4382c7a18d-yjhxtv@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) client-ip=185.136.64.225; Received: by mta-64-225.siemens.flowmailer.net with ESMTPSA id 20240719163844e28a1b2b4382c7a18d for ; Fri, 19 Jul 2024 18:38:44 +0200 From: "'Jan Kiszka' via isar-users" To: isar-users Cc: Silvano Cirujano-Cuesta , Benedikt Niedermayr , Felix Moessbauer Subject: [PATCH v4 1/5] Introduce fetcher from container registries Date: Fri, 19 Jul 2024 18:38:39 +0200 Message-ID: <82da88bf02bf928d8807bc93bfb5fcdeece1f558.1721407122.git.jan.kiszka@siemens.com> In-Reply-To: References: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=jbgFU5+C; spf=pass (google.com: domain of fm-294854-20240719163844e28a1b2b4382c7a18d-yjhxtv@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-294854-20240719163844e28a1b2b4382c7a18d-YjHXtv@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Content-Type: text/plain; charset="UTF-8" Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: Wzkp+hYzdmUu From: Jan Kiszka This bitbake fetcher allows to pull container images from registries, store them in the download cache and transfer them into the workdir of recipes requesting the image. The format of the URL is docker://[/];digest=sha256:...[;tag=] Fetching without digest is supported but will cause a warning, just like downloading via wget without a checksum. If tag is left out, "latest" is used. In case a multi-arch image is specified, the fetcher will only pull for the package architecture of the requesting recipe. The image is stored compressed in docker-archive format and, wherever possible, hard-linked from DL_DIR to WORKDIR. Future versions may also introduce full unpacking of the fetched container layers in workdir if use cases come up. Signed-off-by: Jan Kiszka --- meta/classes/dpkg-base.bbclass | 6 +++ meta/lib/container_fetcher.py | 86 ++++++++++++++++++++++++++++++++++ 2 files changed, 92 insertions(+) create mode 100644 meta/lib/container_fetcher.py diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index 789d6c74..d90b32a9 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -98,6 +98,12 @@ python() { if len(d.getVar('SRC_APT').strip()) > 0: bb.build.addtask('apt_unpack', 'do_patch', '', d) bb.build.addtask('cleanall_apt', 'do_cleanall', '', d) + + # container docker fetcher + import container_fetcher + from bb.fetch2 import methods + + methods.append(container_fetcher.Container()) } do_apt_fetch() { diff --git a/meta/lib/container_fetcher.py b/meta/lib/container_fetcher.py new file mode 100644 index 00000000..0d659154 --- /dev/null +++ b/meta/lib/container_fetcher.py @@ -0,0 +1,86 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +import oe.path +import os +import tempfile +from bb.fetch2 import FetchMethod +from bb.fetch2 import logger +from bb.fetch2 import MissingChecksumEvent +from bb.fetch2 import NoChecksumError +from bb.fetch2 import runfetchcmd + +class Container(FetchMethod): + def supports(self, ud, d): + return ud.type in ['docker'] + + def urldata_init(self, ud, d): + ud.tag = "latest" + if "tag" in ud.parm: + ud.tag = ud.parm["tag"] + + ud.digest = None + if "digest" in ud.parm: + ud.digest = ud.parm["digest"] + + ud.arch = d.getVar('PACKAGE_ARCH') + ud.variant = None + if ud.arch == "armhf": + ud.arch = "arm" + ud.variant = "v7" + elif ud.arch == "armel": + ud.arch = "arm" + ud.variant = "v6" + + ud.container_name = ud.host + (ud.path if ud.path != "/" else "") + ud.container_src = ud.container_name + \ + ("@" + ud.digest if ud.digest else ":" + ud.tag) + ud.localname = ud.container_name.replace('/', '.') + ud.localfile = "container-images/" + ud.arch + "/" + \ + (ud.variant + "/" if ud.variant else "") + ud.localname + \ + "_" + (ud.digest.replace(":", "-") if ud.digest else ud.tag) + \ + ".zst" + + def download(self, ud, d): + tarball = ud.localfile[:-len('.zst')] + with tempfile.TemporaryDirectory(dir=d.getVar('DL_DIR')) as tmpdir: + # Take a two steps for downloading into a docker archive because + # not all source may have the required Docker schema 2 manifest. + runfetchcmd("skopeo copy --preserve-digests " + \ + f"--override-arch {ud.arch} " + \ + (f"--override-variant {ud.variant} " if ud.variant else "") + \ + f"docker://{ud.container_src} dir:{tmpdir}", d) + runfetchcmd(f"skopeo copy dir:{tmpdir} " + \ + f"docker-archive:{tarball}:{ud.container_name}:{ud.tag}", d) + zstd_defaults = d.getVar('ZSTD_DEFAULTS') + runfetchcmd(f"zstd -f --rm {zstd_defaults} {tarball}", d) + + if ud.digest: + return + + checksum = bb.utils.sha256_file(ud.localpath + "/manifest.json") + checksum_line = f"SRC_URI = \"{ud.url};digest=sha256:{checksum}\"" + + strict = d.getVar("BB_STRICT_CHECKSUM") or "0" + + # If strict checking enabled and neither sum defined, raise error + if strict == "1": + raise NoChecksumError(checksum_line) + + checksum_event = {"sha256sum": checksum} + bb.event.fire(MissingChecksumEvent(ud.url, **checksum_event), d) + + if strict == "ignore": + return + + # Log missing digest so user can more easily add it + logger.warning( + f"Missing checksum for '{ud.localpath}', consider using this " \ + f"SRC_URI in the recipe:\n{checksum_line}") + + def unpack(self, ud, rootdir, d): + image_file = ud.localname + ":" + ud.tag + ".zst" + oe.path.remove(rootdir + "/" + image_file) + oe.path.copyhardlink(ud.localpath, rootdir + "/" + image_file) -- 2.43.0 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/82da88bf02bf928d8807bc93bfb5fcdeece1f558.1721407122.git.jan.kiszka%40siemens.com.