From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Mon, 15 Jul 2024 12:08:43 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f191.google.com (mail-lj1-f191.google.com [209.85.208.191]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46FA8g3W005839 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 15 Jul 2024 12:08:42 +0200 Received: by mail-lj1-f191.google.com with SMTP id 38308e7fff4ca-2ee81ff717fsf45043911fa.3 for ; Mon, 15 Jul 2024 03:08:42 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721038117; cv=pass; d=google.com; s=arc-20160816; b=L7ayvJIr6RATjXIi+SYm1WeQjPdQ8WANjw3Qsj9fBq9L/dr6la7aW7XSk/b3ngIgDZ NtF8qun1N55hkySGGTXYCtulPO2DlZWgwFBzJDKJ+o0aT2K5J8NXMtXUH1P27t1URqCj BwzWeAuzijwPif9kSkqXZkLZKqnldgBpPkfv2r6KGbY7AQIYEgRTtGjLRu8nfI7JY1wp QCu4TN/N+96NPtkGhiBaFlgMUHtnrXuMu5qw74SC8ACp2IHT2WZJ5ggfM5LwM7vVCf4b yly3E8LKmR0eiuw/05yINt5JKV7w76ZV+lZytmeGxkKf20TOzQEyKiV6NVTgSZmwwVIv +2iA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=eMj33Q6H69Xqo1ZWMiIHEwlSOEWGrXiiAt2LdIn1FRY=; fh=ensxGhSVkX8go+Rl4xPv+rENqmhIovtxRkY+ROkmmoM=; b=GSw3sfMIr68S8P2neIMaQw/LgTEAHApL+bgFCgwo2OyF4sh+v7SoRfFmcYSxjSLlrg Y8trUskt8ksaUI0yPn8B1g9OVESwz+bqFS2/zGHnU9j8rSatfgd8UrsFWQ6dwYCHeiGv Nsf1XjUkxHRu+AizpwW/w196yO+irzhak6E+PcMSQyIHKFD6dgYl/EX9UtHoT6GLx2tY zJ1CveY/C43a/4a22tbomMmaXC9CjVHKrLBZu/MleYxj4KFFeFPjhIyWljD+MLCMoXm7 ylRQX/VX9gZGCeuueMZg9AjOSFrQ6DArgolINUB1r3m/hO05ehz5YS2sFBFh5A6SjBlV +m9g==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=NEuTmpTV; spf=pass (google.com: domain of fm-294854-202407151008334113b2dafda22e9ba9-_88zb_@rts-flowmailer.siemens.com designates 185.136.65.228 as permitted sender) smtp.mailfrom=fm-294854-202407151008334113b2dafda22e9ba9-_88Zb_@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721038117; x=1721642917; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=eMj33Q6H69Xqo1ZWMiIHEwlSOEWGrXiiAt2LdIn1FRY=; b=PaisrenUlKjSQ2yuXaDQMj/HJErG/DQJghiWPy0z29ZUGqC1MEwvlqk1e9fJOcoksw OnV9m02Ffreft7n28amhGaJGGk0u1CyCatPGR3ncSaTCoC7LHLR4Q87ih4Z3TD6NyOO1 JnEd3G0U6I+nE7BJZQEk0ZGIWe7g8LvdvTPlFj/7Yy9UvEdUMtRJvnCl2fa2RGLTEXWQ WGaS6YIj4TZhm8CHCAw2QRYF5hkUQW3xmcX6/oY0jQvVJMYCx6V2U8NheJwrjocEMcEU 0/4riAo6mFv8Ex+meHSrD1pV3yjfNF868G8YQOtoUOoFWeicaf14prQRP2Em1M1BbtQi uiAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721038117; x=1721642917; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=eMj33Q6H69Xqo1ZWMiIHEwlSOEWGrXiiAt2LdIn1FRY=; b=r4LShH6x3uFMQVywDvqJk2u8hiMSvMOaRzpglEWbPZekY5Exl3NROOIBu1mRVA7cEW dKCyTogckVCzaQ/+aUsC37QAavKk/9kXQO96v31RLD+w4KaA4ES4PHNNqEFpx73fQpF7 9TEVwjO64fRDNTEbDQESuUJ8cjVw7rxPYTvadkKSS6qjrcxdaY0qz0YfdzAPTlmThStI Dw29Uw7Wdxc5wN5taiOiZuRqlC2JtuEXFRx+RNSeFxxPdNf7VmAwSsNcntvGjxJuvCC1 oM2+Ah6ESuxRbH8qwSlu91IRBAbvuvajBomcDWJyHsduvStMEIVLVUya7jgMxADNL1AO rHCw== X-Forwarded-Encrypted: i=2; AJvYcCWWR/h2UeE2S1YeidpkWrM7K19IIm8V3MlpqXZPb3u7HSYMfYYL6Wmyc7O73CtwJfewwKbNTvkYDucvbPuq5BfjNbg= X-Gm-Message-State: AOJu0YxPaTWQJL0iBziHZGiCNU7KEsgkIJhCEf2x3QtCRy7K4kgpBJBH i/jp5eKfxR0TK6OBpES5kpBX8t5G1ye87Q3p9Q+XPClyo3bTdUZI X-Google-Smtp-Source: AGHT+IFZUg67Cmcta18vrBLhsdSqhwJpD8LW8jaUlnneW5JcfMO0tOKf6DhcZQ6SlvhIsnwkfF5tGQ== X-Received: by 2002:a2e:8081:0:b0:2ee:83f7:8b34 with SMTP id 38308e7fff4ca-2eeb30ba05cmr140867551fa.8.1721038116249; Mon, 15 Jul 2024 03:08:36 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:651c:489:b0:2ee:7a6c:fcba with SMTP id 38308e7fff4ca-2eec938cc1fls18789091fa.1.-pod-prod-08-eu; Mon, 15 Jul 2024 03:08:34 -0700 (PDT) X-Received: by 2002:a2e:911a:0:b0:2eb:2e0b:72c with SMTP id 38308e7fff4ca-2eeb30dc843mr140984731fa.16.1721038113883; Mon, 15 Jul 2024 03:08:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721038113; cv=none; d=google.com; s=arc-20160816; b=JU4wn0VRM5hy+yIn4zvGzsxkvRsgoHtT/q+G4apQH+GK66rwS9HzPanm5gjPUsdKQx 5UUWpLBwEVTSvC+yPVHUbTKfz/BFw1FacPWufxGOxPM1WPBO0g2PGNMbdkLNTFiT1jkT y/2qAajUQkpKVfz+MeBmJ71dLnvJnjjyfwtksW+5rblzq9hwBSQaKfAMp7MzcK/ey6jy D5EpNu9TTp9CEq88/xFhODnVomDfcEa0eae9IwKkOqvnESFqregVlh1rl5eMuHGva9tO C9HVBgk9ljSqDbVu9p/vyArtOS/1chLD3iefsxFQWwI0Yp98YEeEmB+OIsu7GAGwPw2v BUKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=qT0qBD6zoZX/tPGQ0GTumBOagxBAE1nu2vnVszdRf/0=; fh=sG8mVX6bNXJXg1RhSNhryk9YKHryCWWF37H72hfFhEU=; b=AE6Fas+i8Fg5x6PpoF4G3P3lo6fGTAZg2OPujtmZa2vcT8q4VJLqHf2pUb46YW572F 3E0/OQsPoCUNoRyv1CtFG5yv0wF+JheNbL5xkTAPca+ZciRNPVj/roZDna0IC3L3jGSK 1N6+v/lEv4aDVRp+/0cyoSqaWolmlPkTYEbfl90FLuO8XgdLtnsjwJdF3BYNMnQE1k/A 9061UVWz9jEDZyU2WAbk0q9HY9/oMjlDAE7tLw+GjUSEad7YTTiKsKJ4OL5JijwI3PwF QGdW5q9jKqPLY8fiXn8UrVX7yMA2cLNRudX9rXYukR8CgNMNrcJ/ns59T8S7WwuijQIl kHCg==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=NEuTmpTV; spf=pass (google.com: domain of fm-294854-202407151008334113b2dafda22e9ba9-_88zb_@rts-flowmailer.siemens.com designates 185.136.65.228 as permitted sender) smtp.mailfrom=fm-294854-202407151008334113b2dafda22e9ba9-_88Zb_@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-65-228.siemens.flowmailer.net (mta-65-228.siemens.flowmailer.net. [185.136.65.228]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-427a5ec6cb7si674365e9.1.2024.07.15.03.08.33 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 15 Jul 2024 03:08:33 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-202407151008334113b2dafda22e9ba9-_88zb_@rts-flowmailer.siemens.com designates 185.136.65.228 as permitted sender) client-ip=185.136.65.228; Received: by mta-65-228.siemens.flowmailer.net with ESMTPSA id 202407151008334113b2dafda22e9ba9 for ; Mon, 15 Jul 2024 12:08:33 +0200 From: "'Jan Kiszka' via isar-users" To: isar-users Cc: Silvano Cirujano-Cuesta , Benedikt Niedermayr , Felix Moessbauer Subject: [PATCH v2 1/5] Introduce fetcher from container registries Date: Mon, 15 Jul 2024 12:08:28 +0200 Message-ID: In-Reply-To: References: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=NEuTmpTV; spf=pass (google.com: domain of fm-294854-202407151008334113b2dafda22e9ba9-_88zb_@rts-flowmailer.siemens.com designates 185.136.65.228 as permitted sender) smtp.mailfrom=fm-294854-202407151008334113b2dafda22e9ba9-_88Zb_@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Content-Type: text/plain; charset="UTF-8" Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: WN88ciimj9uF From: Jan Kiszka This bitbake fetcher allows to pull container images from registries, store them in the download cache and transfer them into the workdir of recipes requesting the image. The format of the URL is docker://[/];digest=sha256:...[;tag=] Fetching without digest is supported but will cause a warning, just like downloading via wget without a checksum. If tag is left out, "latest" is used. The fetcher will try to pull all available variants of a multi-arch image. If this is not needed, you can also directly specify the image digest of a specific architecture. Future versions may also introduce full unpacking of the fetched container layers in workdir if use cases come up. Signed-off-by: Jan Kiszka --- meta/classes/dpkg-base.bbclass | 6 +++ meta/lib/container_fetcher.py | 90 ++++++++++++++++++++++++++++++++++ 2 files changed, 96 insertions(+) create mode 100644 meta/lib/container_fetcher.py diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index 789d6c74..d90b32a9 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -98,6 +98,12 @@ python() { if len(d.getVar('SRC_APT').strip()) > 0: bb.build.addtask('apt_unpack', 'do_patch', '', d) bb.build.addtask('cleanall_apt', 'do_cleanall', '', d) + + # container docker fetcher + import container_fetcher + from bb.fetch2 import methods + + methods.append(container_fetcher.Container()) } do_apt_fetch() { diff --git a/meta/lib/container_fetcher.py b/meta/lib/container_fetcher.py new file mode 100644 index 00000000..8513e246 --- /dev/null +++ b/meta/lib/container_fetcher.py @@ -0,0 +1,90 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +import os +import re +from bb.fetch2 import FetchMethod +from bb.fetch2 import logger +from bb.fetch2 import MissingChecksumEvent +from bb.fetch2 import NoChecksumError +from bb.fetch2 import runfetchcmd +from bb.progress import LineFilterProgressHandler + +class SkopeoProgressHandler(LineFilterProgressHandler): + def __init__(self, d): + super(SkopeoProgressHandler, self).__init__(d) + self._fire_progress(0) + + def writeline(self, line): + match = re.findall(r'^Copying image .*\(([0-9]+/[0-9]+)\)$', line) + if match: + state = match[0].split('/') + progress = (int(state[0]) * 100) / int(state[1]) + self.update(progress) + return True + + +class Container(FetchMethod): + def supports(self, ud, d): + return ud.type in ['docker'] + + def urldata_init(self, ud, d): + ud.tag = "latest" + if "tag" in ud.parm: + ud.tag = ud.parm["tag"] + + ud.digest = None + if "digest" in ud.parm: + ud.digest = ud.parm["digest"] + + container_name = ud.host + (ud.path if ud.path != "/" else "") + ud.container_src = container_name + \ + ("@" + ud.digest if ud.digest else ":" + ud.tag) + ud.localname = container_name.replace('/', '.') + ud.localfile = "container-images/" + ud.localname + \ + "_" + (ud.digest.replace(":", "-") if ud.digest else ud.tag) + + def download(self, ud, d): + progresshandler = SkopeoProgressHandler(d) + runfetchcmd(f"skopeo copy --preserve-digests --all docker://{ud.container_src} dir:{ud.localfile}", + d, log=progresshandler) + + if ud.digest: + return + + checksum = bb.utils.sha256_file(ud.localpath + "/manifest.json") + checksum_line = f"SRC_URI = \"{ud.url};digest=sha256:{checksum}\"" + + strict = d.getVar("BB_STRICT_CHECKSUM") or "0" + + # If strict checking enabled and neither sum defined, raise error + if strict == "1": + raise NoChecksumError(checksum_line) + + checksum_event = {"sha256sum": checksum} + bb.event.fire(MissingChecksumEvent(ud.url, **checksum_event), d) + + if strict == "ignore": + return + + # Log missing digest so user can more easily add it + logger.warning( + f"Missing checksum for '{ud.localpath}', consider using this " \ + f"SRC_URI in the recipe:\n{checksum_line}") + + def unpack(self, ud, rootdir, d): + arch = d.getVar('PACKAGE_ARCH') + variant_opt = "" + if arch == "armhf": + arch = "arm" + variant_opt = "--override-variant v7" + elif arch == "armel": + arch = "arm" + variant_opt = "--override-variant v6" + runfetchcmd(f"skopeo --override-arch {arch} {variant_opt} " \ + f"copy dir:{ud.localpath} dir:{rootdir + '/' + ud.localname}", d) + + def clean(self, ud, d): + bb.utils.remove(ud.localpath, recurse=True) -- 2.43.0 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/b0493bec83353990199120a5fcf43676e995bba8.1721038111.git.jan.kiszka%40siemens.com.