From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6925703954041929728 X-Received: by 2002:a2e:8887:: with SMTP id k7mr3443014lji.24.1612549221528; Fri, 05 Feb 2021 10:20:21 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:8745:: with SMTP id q5ls1842345ljj.6.gmail; Fri, 05 Feb 2021 10:20:20 -0800 (PST) X-Google-Smtp-Source: ABdhPJyDMXtq8dtfSRoMz4H9jhxCpvmdyVV3rjISj7KZus0EIDwUUg9qz45g/uxwMmSDwUhGKNwH X-Received: by 2002:a2e:3507:: with SMTP id z7mr3428762ljz.428.1612549220249; Fri, 05 Feb 2021 10:20:20 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1612549220; cv=none; d=google.com; s=arc-20160816; b=HY/LwH0lMciov/DUpvYMRL4hBE0GlGFGg/UzodLn4THIMh7FpKVenF1Jwir3hDljtm kDeF6uN0c3SrDdffHKHpK8yDmhyY86rj/8ru+YrASX+9/DD9shgcdCAiYNF3yHRd9Z+r +BSfFbJouaRY4Ivme3o0qxagzCn6Z/PNE+DmZTd6k1cPKDFKNxHuVKN940v2fu/Xi9DF tS806gbR7r2o1BCSS6rqQ1ZX41ayWymtp1X4QWnPrInjBuDSLrOUF/SyFkGflC2TEi+x AvUQnNgx6oqGY03KPaGGSLzRZzU92r3lRJO+hW9zrK+O5iEuTV7YNDphaIRzyjOCuT1s vm0A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject; bh=aDXbeCSnOO9S+vMNZ1fErkJp4cgVUHCy0cADg0RL0Lo=; b=S5+s0ZqJPpgTmSWQfAGHMpoqSjnZMPM0CC7VRKIqDodYWuPVMDLTQBGEYOkfz6VeGJ pITU1NmuDqvumeIFCSlPdR3PaikBMF7lqp2rPKCiK0YMTlN0Gk8uaEhj77oUIYXEs3Nh daMX4x5yLl0H5yEhkSv95K/Ha5TqVUnESbqC8C8/6KdnLjvKiIrUzyxDIL0MwtZdgkJp VJrr0lchWxzGQPPD3pD0JEQ0yrX/m4gf17W6h24kk85h1e8LXrMOG5X+rSBQwi8jA+5z D+d8MbumfsAmx/013grOn5ceYbJu/s4C2y9GYXQlyvEkZJZnE1Kt0dCSEIzgZLRqOJuw UgHw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id f18si565017ljj.1.2021.02.05.10.20.20 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 05 Feb 2021 10:20:20 -0800 (PST) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id 115IKJxa020822 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Fri, 5 Feb 2021 19:20:19 +0100 Received: from [167.87.72.79] ([167.87.72.79]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id 115IKIbM030538; Fri, 5 Feb 2021 19:20:18 +0100 Subject: Re: [PATCH 1/2] sdk: support creation of container image To: Silvano Cirujano Cuesta , isar-users@googlegroups.com References: <20210205090827.17788-1-silvano.cirujano-cuesta@siemens.com> <20210205090827.17788-2-silvano.cirujano-cuesta@siemens.com> <1bf47211-0313-48a9-00d8-442e6f9942ae@siemens.com> <62b4993c-0a7d-f29d-9c65-978936fd8e7c@siemens.com> <724da05d-7eeb-73de-05a4-5effab51086f@siemens.com> From: Jan Kiszka Message-ID: <34655006-92e7-f656-ae98-7a667b9c5a89@siemens.com> Date: Fri, 5 Feb 2021 19:20:18 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: ezERan7Hsjao On 05.02.21 18:52, Silvano Cirujano Cuesta wrote: > > On 05/02/2021 12:29, Jan Kiszka wrote: >> On 05.02.21 12:24, Silvano Cirujano Cuesta wrote: >>> On 05/02/2021 12:07, Jan Kiszka wrote: >>>> On 05.02.21 10:08, [ext] Silvano Cirujano Cuesta wrote: >>>>> Extend task "populate_sdk" to support the creation of a container image >>>>> containing the SDK. >>>>> >>>>> Signed-off-by: Silvano Cirujano Cuesta >>>>> --- >>>>> meta/classes/image-sdk-extension.bbclass | 104 +++++++++++++++++++++-- >>>>> 1 file changed, 97 insertions(+), 7 deletions(-) >>>>> >>>>> diff --git a/meta/classes/image-sdk-extension.bbclass b/meta/classes/image-sdk-extension.bbclass >>>>> index a8c708a..082b16d 100644 >>>>> --- a/meta/classes/image-sdk-extension.bbclass >>>>> +++ b/meta/classes/image-sdk-extension.bbclass >>>>> @@ -6,10 +6,81 @@ >>>>> # This class extends the image.bbclass to supply the creation of a sdk >>>>> >>>>> SDK_INCLUDE_ISAR_APT ?= "0" >>>>> +SDK_FORMATS ?= "tar-xz" >>>>> + >>>>> +sdk_tar_xz() { >>>>> + # Copy mount_chroot.sh for convenience >>>>> + sudo cp ${SCRIPTSDIR}/mount_chroot.sh ${SDKCHROOT_DIR} >>>>> + >>>>> + # Create SDK archive >>>>> + cd -P ${SDKCHROOT_DIR}/.. >>>>> + sudo tar --transform="s|^rootfs|sdk-${DISTRO}-${DISTRO_ARCH}|" \ >>>>> + -c rootfs | xz -T0 > ${DEPLOY_DIR_IMAGE}/sdk-${DISTRO}-${DISTRO_ARCH}.tar.xz >>>>> + bbnote "SDK rootfs available in ${DEPLOY_DIR_IMAGE}/sdk-${DISTRO}-${DISTRO_ARCH}.tar.xz" >>>>> +} >>>>> + >>>>> +sdk_container_images() { >>>>> + local cmd="/bin/dash" >>>>> + local empty_tag="empty" >>>>> + local full_tag="latest" >>>>> + local oci_img_dir="${WORKDIR}/oci-image" >>>>> + local sdk_container_formats="$1" >>>>> + >>>>> + # prepare OCI container image skeleton >>>>> + sudo umoci init --layout "${oci_img_dir}" >>>>> + sudo umoci new --image "${oci_img_dir}:${empty_tag}" >>>>> + sudo umoci config --image "${oci_img_dir}:${empty_tag}" \ >>>>> + --config.cmd="${cmd}" >>>>> + sudo umoci unpack --image "${oci_img_dir}:${empty_tag}" \ >>>>> + "${oci_img_dir}_unpacked" >>>>> + >>>>> + # add SDK root filesystem as the flesh of the skeleton >>>>> + sudo cp -a "${SDKCHROOT_DIR}"/* "${oci_img_dir}_unpacked/rootfs/" >>>>> + >>>>> + # pack container image >>>>> + sudo umoci repack --image "${oci_img_dir}:${full_tag}" \ >>>>> + "${oci_img_dir}_unpacked" >>>>> + sudo umoci remove --image "${oci_img_dir}:${empty_tag}" >>>>> + sudo rm -rf "${oci_img_dir}_unpacked" >>>>> + >>>>> + # no root needed anymore >>>>> + sudo chown --recursive $(id -u):$(id -g) "${oci_img_dir}" >>>>> + >>>>> + # convert the OCI container image to the desired format >>>>> + sdk_id="sdk-${DISTRO}-${DISTRO_ARCH}" >>>>> + image_name="isar-${sdk_id}" >>>>> + image_archive="${DEPLOY_DIR_IMAGE}/${sdk_id}-${sdk_format}.tar" >>>>> + for sdk_format in ${sdk_container_formats} ; do >>>>> + case "${sdk_format}" in >>>>> + "docker-archive" | "oci-archive") >>>>> + if [ "${sdk_format}" = "oci-archive" ] ; then >>>>> + target="${sdk_format}:${image_archive}:latest" >>>>> + else >>>>> + target="${sdk_format}:${image_archive}:${image_name}:latest" >>>>> + fi >>>>> + skopeo --insecure-policy copy \ >>>>> + "oci:${oci_img_dir}:${full_tag}" "${target}" >>>>> + xz -T0 "${image_archive}" >>>>> + bbnote "Containerized SDK available in ${image_archive}.xz" >>>>> + ;; >>>>> + "oci") >>>>> + tar --create --xz --directory "${oci_img_dir}" \ >>>>> + --file "${image_archive}.xz" . >>>>> + bbnote "Containerized SDK available in ${image_archive}.xz" >>>>> + ;; >>>>> + "docker-daemon" | "containers-storage") >>>>> + skopeo --insecure-policy copy \ >>>>> + "oci:${oci_img_dir}:${full_tag}" \ >>>>> + "${sdk_format}:${image_name}:latest" >>>>> + bbnote "Containerized SDK available in ${sdk_format} as '${image_name}:latest'" >>>>> + ;; >>>>> + esac >>>>> + done >>>>> +} >>>>> >>>>> do_populate_sdk[stamp-extra-info] = "${DISTRO}-${MACHINE}" >>>>> do_populate_sdk[depends] = "sdkchroot:do_build" >>>>> -do_populate_sdk[vardeps] += "SDK_INCLUDE_ISAR_APT" >>>>> +do_populate_sdk[vardeps] += "SDK_INCLUDE_ISAR_APT SDK_FORMATS" >>>>> do_populate_sdk() { >>>>> if [ "${SDK_INCLUDE_ISAR_APT}" = "1" ]; then >>>>> # Copy isar-apt with deployed Isar packages >>>>> @@ -48,12 +119,31 @@ do_populate_sdk() { >>>>> done >>>>> done >>>>> >>>>> - # Copy mount_chroot.sh for convenience >>>>> - sudo cp ${SCRIPTSDIR}/mount_chroot.sh ${SDKCHROOT_DIR} >>>>> + # separate SDK formats: TAR and container formats >>>>> + container_formats="" >>>>> + for sdk_format in ${SDK_FORMATS} ; do >>>>> + case ${sdk_format} in >>>>> + "tar-xz") >>>>> + sdk_tar_xz >>>>> + ;; >>>>> + "docker-archive" | "oci" | "oci-archive") >>>>> + container_formats="${container_formats} ${sdk_format}" >>>>> + ;; >>>>> + "docker-daemon" | "containers-storage") >>>>> + if [ -f /.dockerenv ] || [ -f /run/.containerenv ] ; then >>>>> + die "Adding the SDK container image to a container runtime (${sdk_format}) not supported if running from a container (e.g. 'kas-container')" >>>>> + fi >>>>> + ;; >>>>> + *) >>>>> + die "unsupported SDK format specified: ${sdk_format}" >>>>> + ;; >>>>> + esac >>>>> + done >>>>> >>>>> - # Create SDK archive >>>>> - cd -P ${SDKCHROOT_DIR}/.. >>>>> - sudo tar --transform="s|^rootfs|sdk-${DISTRO}-${DISTRO_ARCH}|" \ >>>>> - -c rootfs | xz -T0 > ${DEPLOY_DIR_IMAGE}/sdk-${DISTRO}-${DISTRO_ARCH}.tar.xz >>>>> + # generate the SDK in all the desired container formats >>>>> + if [ -n "${container_formats}" ] ; then >>>>> + bbnote "Generating SDK container in${container_formats} format" >>>>> + sdk_container_images "${container_formats}" >>>>> + fi >>>>> } >>>>> addtask populate_sdk after do_rootfs >>>>> >>>> How much of this would be reusable of generating a container from a >>>> target rootfs? We should avoid shuffling code around if we can already >>>> line things up nicely while introducing it. >>> With that reuse in mind I can refactor the code to have a class that can be reused for both SDK and target rootfs. It's not a big deal. >>> >>> Is it somehow related to the huge "some image classes" thread that is active right now? >>> >> Not directly related, but container images would fall into that same >> category. > > What do you have in mind? How should I structure the functions, classes, tasks,...? > > In a separate bbclass providing only generic functions that can containerize whatever rootfs (e.g. target) they're given (currently only for the SDK calling them from "populate_sdk")? Similar to "debianize.bbclass". Something I can do easily. > > In a separate bbclass following the pattern of the other "-img.bclass" (e.g. "container-img.bbclass")? I mean providing not only functions even new tasks. Would be harder to accomplish. > Yes, that is what I was thinking of (and that user on our internal issue tracker as well, if you recall): Drop-in replacement for existing target image classes, e.g. selected via IMAGE_TYPE. > Are the "-img.bbclass" classes only for target images? I wonder why "targz-img.bbclass" doesn't take care of "packaging" also the SDK. Not because of code reuse (too short code, to many differences on the TAR call), but for logical consistency. So far, this is the logic. I didn't dig into the commonalities between the sdk tarball and that - later added - targz-img. Maybe that is worth to refactor when adding something like the container formats. Jan -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux