From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Tue, 09 Jul 2024 19:31:50 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lf1-f55.google.com (mail-lf1-f55.google.com [209.85.167.55]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 469HVne6006531 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 9 Jul 2024 19:31:50 +0200 Received: by mail-lf1-f55.google.com with SMTP id 2adb3069b0e04-52e960523ecsf5365744e87.1 for ; Tue, 09 Jul 2024 10:31:50 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1720546304; cv=pass; d=google.com; s=arc-20160816; b=ReFrcW9sXmQJ8oGGOV+THye+BvVNrpCEzFtywvEV67Ml6WY9GSuxzFzOPmqiRhnHSA yFJi8N22wsb1Wf966KBLSDpUQNDKHeshbacnP7dln6yiMxvgocnSBXkdFWdGtrKriEKC yo6ZC2EiwdQ9FDObkjL3kWqpH6HrwJgBYhhoi8B8CNWqF3Pq0/p/wMZMKDyUY/oz3wG3 Vj24YlpUgCyyscmPzNbuigdB/yJha1QKf28Sjv7CmDxDFzYl56XHAZltwLPBdQzGE6H3 QvyNpVsuKe1SKaYi/qPHlf2ZN+WcRhe33DbwmMMI9TVrbX5yf9nHT83F6U1pcXPBA6sC 3Qgw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=wUR46tngSfB5cZc8nNKXaUBTvDRWVT0hOZzOppbQG3o=; fh=qSzgsDhavCE53sz+5X0WtJqk6J93SdMgv6S8fEh3Pqw=; b=Dp/lrMRJQfQrTFOOP+BxGNYzBd6bNMl0IHtbrEH3ZkkH5yL7pZjfZR1HYPV/fg3azg mssTOD/+VFIPuRD1uiOLB3op0VhORctV3bI7LgFS8wHgwLNssZ9gqdaccvLrW9l/0lC3 yaHE5FOXtsk94EbbBglW8uQcJ9IaW/tKFjY7TPxY208xCo44cLFeokVAMqYgPW+Nonxa Mjhyyfv89bRfow2YpCB9cEs4sBPgrxQOUjB4YKd5gx+Vvyg/CanEQcH1GMlDHRgI+KOY VzseDcG04ZFmT4LDWNSwCyrscsHXFG9AR1sWRWslGlkNovgCuTwI4oVaq/QQDPPrhd9a 3Npw==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=L3VrYHtm; spf=pass (google.com: domain of fm-294854-2024070917314078e9b583f9bda058c6-fiuspa@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-294854-2024070917314078e9b583f9bda058c6-FIUSpa@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1720546304; x=1721151104; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:to:from :from:to:cc:subject:date:message-id:reply-to; bh=wUR46tngSfB5cZc8nNKXaUBTvDRWVT0hOZzOppbQG3o=; b=lzobixiV5rrc/A2TVPp+KeGEwpGtSgoK061g3O4dja6iaA0xNG2Dt4HkBOM6Ndiv9G MHJKUyoe/Y9m+ITlHttYR6qRINT8LtfBJIfyWj2EsPbuGzcNWXa4wmKZUvnMRrvD46/+ QgcUGS4bKo2AXSM/3cCImAPgYFqv2qUSSe0ggNGsMl3O1Ca/Lx+VRgSFsk9A6VYbhuvO vb81gq/DMgz8s+m0366bZvgN/ffiISM8CldMO9eucIf1rDTngyATS+vXj3Uf4yjnw+B9 ai3K9MFHbkdBsp4CzWJChBNcYQ6/eEeOR1l5cw2TSFnp3SO1KLkzGCpXlN+jyVfjNiRd 3HBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720546304; x=1721151104; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:to:from :x-beenthere:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=wUR46tngSfB5cZc8nNKXaUBTvDRWVT0hOZzOppbQG3o=; b=g9wUIqicO7qVFN1wsD55kIprDTszJPFxkEF0XcrilA/mb9B5+ch3Z9akMg5v5TozGQ BRt8VvPyxKL3B4NJ7DavR/MU9A2PsGIMRQ9GSnPkvA6zgri9qIZfgLBBB0+5DkjfaKeP DP6Vilnbbli/IIOF1QNQPyC6jX3igMFUsCrWKtOW0q2Q9VptI4LFh/rpQbCaXnLS4/RO +wscIU51cOBhKDLiT6H7/GDuGMh2fRlcHucz3dPDLTRZ3NHzo7uTXZYs46gFQomE2oyI Wj8la3BKXijiXy3HUnYqspuvRWsllMghkZq/PXQ3UKcLcrcocZs+a+nykq7cVrDzEAAa h8iA== X-Forwarded-Encrypted: i=2; AJvYcCW77DJJGARh8X7BO1VxQKcFhEZO0MTzNMXpcMcKvePE6KQ02BZvZ9gT6/L/g+AJ1c57Yo6ibr8lMc7/se3TLsZyy9M= X-Gm-Message-State: AOJu0YzCEHX2gspfxW0aqpAeBT6FC3HzL3qpyAq/T2iOgjTXNejhKp7m 5xtA473su6Vt/MLNYUFy6E3CE22BICwDzuaDu/X3eh2QpNjf7qrf X-Google-Smtp-Source: AGHT+IFbzF9vTf5QEz/NXr6d4bHeiD8Y5qbN/UinPZKyVhF9jEHU8xhSb+v26eFdi1D/nf2tP/6MNA== X-Received: by 2002:a05:6512:3b87:b0:52e:9d2c:1c81 with SMTP id 2adb3069b0e04-52eb9990ccamr2336007e87.13.1720546303719; Tue, 09 Jul 2024 10:31:43 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6512:31d3:b0:52c:a105:90ee with SMTP id 2adb3069b0e04-52e9f301e67ls1488533e87.1.-pod-prod-01-eu; Tue, 09 Jul 2024 10:31:41 -0700 (PDT) X-Received: by 2002:a05:6512:3d8f:b0:52e:7688:2817 with SMTP id 2adb3069b0e04-52eb9994ab3mr1863067e87.17.1720546301347; Tue, 09 Jul 2024 10:31:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1720546301; cv=none; d=google.com; s=arc-20160816; b=LNuVddjc93aBv7Ulep3tn4Z7qY/62ThyFWDJ05Z1Kh/rqH1CVe/LRU9fLrrG63LTVo pw0F40l8J3Uxy8lk0x9MTQfJNM6mzW28zVG7Stn2Vp0ep+qUE0XazlL1hjCDoWSU/VRr FuYGUzwJZPcFbVRicFGq6M/xnpMUK6mdPV2DO5nP7Tz9mPuV3W+MTekN12BucEgyu9gM acJe/2iDo+4pslaNuFRB0sv+XCd7kCKfQ1TEZc/7REw7xhc9s5z+lwngrwqDkZN5d4z5 LC7W+CryOWkzyWTGSAqoG0+C0gxlYeK4KxT87wWwA2F4vAsNYQqP64ZYDsdFyBN2hCIb 6hiw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:to:from:dkim-signature; bh=fanyJ2CdPImoQzmlj8y6E58TDb+QAfcBjuwhNoStB48=; fh=hOBXq6nisExkIRIwN8IDOmOCLKwX8PM0ANlxQbNMEqA=; b=hGom/G19dkdpbmfRCVWh3ft5rZjChIdc6XNke4ElhMcrr23MkYGtipEb196XjXle17 gfqMqv9gjyF93zFq32Nuvw2oRmAvUWj8t5qpgP1uORzdJKoOHM1TIq/baOIA28MPrkxu Rfy0qZkSuwNh7NiWKzpjKHQkaQluX5i3vl8tfXuv7ZaDiGOevg4oYfXYsNtCjRlSpS+g 4wtIiM88+TOi1g+wL4lXQPr2gCaNIAsOws2uhpP9SulP6XBP0PIx/nJfiN5U6kMqN/Rj KYnpENROYRqJO0xo6aUD9O/1HIDMSmzkp5YT0JPN8XWRhmO5d2M8Jz2CGK7NQOlQ72cc A6hg==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=L3VrYHtm; spf=pass (google.com: domain of fm-294854-2024070917314078e9b583f9bda058c6-fiuspa@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-294854-2024070917314078e9b583f9bda058c6-FIUSpa@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-64-225.siemens.flowmailer.net (mta-64-225.siemens.flowmailer.net. [185.136.64.225]) by gmr-mx.google.com with ESMTPS id 2adb3069b0e04-52eb8e5237bsi89199e87.7.2024.07.09.10.31.41 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Jul 2024 10:31:41 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-294854-2024070917314078e9b583f9bda058c6-fiuspa@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) client-ip=185.136.64.225; Received: by mta-64-225.siemens.flowmailer.net with ESMTPSA id 2024070917314078e9b583f9bda058c6 for ; Tue, 09 Jul 2024 19:31:40 +0200 From: "'Jan Kiszka' via isar-users" To: isar-users Subject: [RFC][PATCH 1/3] Introduce fetcher from container registries Date: Tue, 9 Jul 2024 19:31:37 +0200 Message-ID: In-Reply-To: References: MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-294854:519-21489:flowmailer X-Original-Sender: jan.kiszka@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=L3VrYHtm; spf=pass (google.com: domain of fm-294854-2024070917314078e9b583f9bda058c6-fiuspa@rts-flowmailer.siemens.com designates 185.136.64.225 as permitted sender) smtp.mailfrom=fm-294854-2024070917314078e9b583f9bda058c6-FIUSpa@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Jan Kiszka Reply-To: Jan Kiszka Content-Type: text/plain; charset="UTF-8" Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: S48yUSXn77US From: Jan Kiszka This bitbake fetcher allows to pull container images from registries, store them in the download cache and transfer them into the workdir of recipes requesting the image. The format of the URL is docker://[/];digest=sha256:...[;tag=] Fetching without digest is supported but will cause a warning, just like downloading via wget without a checksum. If tag is left out, "latest" is used. The fetcher will try to pull all available variants of a multi-arch image. If this is not needed, you can also directly specify the image digest of a specific architecture. Future versions may also introduce full unpacking of the fetched container layers in workdir if use case come up. Signed-off-by: Jan Kiszka --- meta/classes/dpkg-base.bbclass | 6 +++ meta/lib/container_fetcher.py | 90 ++++++++++++++++++++++++++++++++++ 2 files changed, 96 insertions(+) create mode 100644 meta/lib/container_fetcher.py diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index 789d6c74..d90b32a9 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -98,6 +98,12 @@ python() { if len(d.getVar('SRC_APT').strip()) > 0: bb.build.addtask('apt_unpack', 'do_patch', '', d) bb.build.addtask('cleanall_apt', 'do_cleanall', '', d) + + # container docker fetcher + import container_fetcher + from bb.fetch2 import methods + + methods.append(container_fetcher.Container()) } do_apt_fetch() { diff --git a/meta/lib/container_fetcher.py b/meta/lib/container_fetcher.py new file mode 100644 index 00000000..8c713795 --- /dev/null +++ b/meta/lib/container_fetcher.py @@ -0,0 +1,90 @@ +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2024 +# +# SPDX-License-Identifier: MIT + +import os +import re +from bb.fetch2 import FetchMethod +from bb.fetch2 import logger +from bb.fetch2 import MissingChecksumEvent +from bb.fetch2 import NoChecksumError +from bb.fetch2 import runfetchcmd +from bb.progress import LineFilterProgressHandler + +class SkopeoProgressHandler(LineFilterProgressHandler): + def __init__(self, d): + super(SkopeoProgressHandler, self).__init__(d) + self._fire_progress(0) + + def writeline(self, line): + match = re.findall(r'^Copying image .*\(([0-9]+/[0-9]+)\)$', line) + if match: + state = match[0].split('/') + progress = (int(state[0]) * 100) / int(state[1]) + self.update(progress) + return True + + +class Container(FetchMethod): + def supports(self, ud, d): + return ud.type in ['docker'] + + def urldata_init(self, ud, d): + ud.tag = "latest" + if "tag" in ud.parm: + ud.tag = ud.parm["tag"] + + ud.digest = None + if "digest" in ud.parm: + ud.digest = ud.parm["digest"] + + container_name = ud.host + (ud.path if ud.path != "/" else "") + ud.container_src = container_name + \ + ("@" + ud.digest if ud.digest else ":" + ud.tag) + ud.localname = container_name.replace('/', '.') + ud.localfile = "container-images/" + ud.localname + \ + "_" + (ud.digest.replace(":", "-") if ud.digest else ud.tag) + + def download(self, ud, d): + progresshandler = SkopeoProgressHandler(d) + runfetchcmd(f"skopeo copy --preserve-digests --all docker://{ud.container_src} dir:{ud.localfile}", + d, log=progresshandler) + + if ud.digest: + return + + checksum = bb.utils.sha256_file(ud.localpath + "/manifest.json") + checksum_line = f"SRC_URI = \"{ud.url};digest=sha256:{checksum}\"" + + strict = d.getVar("BB_STRICT_CHECKSUM") or "0" + + # If strict checking enabled and neither sum defined, raise error + if strict == "1": + raise NoChecksumError(checksum_line) + + checksum_event = {"sha256sum": checksum} + bb.event.fire(MissingChecksumEvent(ud.url, **checksum_event), d) + + if strict == "ignore": + return + + # Log missing digest so user can more easily add it + logger.warning( + f"Missing checksum for '{ud.localpath}', consider using this " \ + f"SRC_URI in the recipe:\n{checksum_line}") + + def unpack(self, ud, rootdir, d): + arch = d.getVar('DISTRO_ARCH') + variant_opt = "" + if arch == "armhf": + arch = "arm" + variant_opt = "--override-variant v7" + elif arch == "armel": + arch = "arm" + variant_opt = "--override-variant v6" + runfetchcmd(f"skopeo --override-arch {arch} {variant_opt} " \ + f"copy dir:{ud.localpath} dir:{rootdir + '/' + ud.localname}", d) + + def clean(self, ud, d): + bb.utils.remove(ud.localpath, recurse=True) -- 2.43.0 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/b2008f2718cc2040403819d0531e8bc2312505b4.1720546299.git.jan.kiszka%40siemens.com.