From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6670845595705409536 X-Received: by 2002:a1c:230e:: with SMTP id j14mr14154827wmj.120.1553584999004; Tue, 26 Mar 2019 00:23:19 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:adf:ec8d:: with SMTP id z13ls149171wrn.3.gmail; Tue, 26 Mar 2019 00:23:18 -0700 (PDT) X-Google-Smtp-Source: APXvYqwo4mJK7LpKy2lnkQ03m2hxBBP0nwR7boTC6LwtC20MFnu62az7RBeQkshQIk00RInkCl9t X-Received: by 2002:adf:f68d:: with SMTP id v13mr20175383wrp.6.1553584998582; Tue, 26 Mar 2019 00:23:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1553584998; cv=none; d=google.com; s=arc-20160816; b=mX3lINHgeXqFdWNuKUYblnE8eaQBwVYeAvi5k97pQ8CujBEnPlTinz5WAQxp9i/4v6 +GgtyvybC3SshIrIIsU+FWor3sgBiPu8ztoXjkglGZ/pX67E58tjdp1DXtwBaS/t/T8O ABoYEqSml9e6ccv62NowDdV4bgTmb/Da2kBu1B0DwxmXffMNPv4mqP7vhozUGVw6PDOw 7bkIiIVpSkVfY4BvTE6q1N1cGE/wUCwo2rqTRjC8P/0zbT0xJwlYIRG2+nYeKMl4mne7 apvA2Noo9Bdr/suFk9kC9vgTVh+QJ8KCfBIM/vVEa/1Sp+ky4JN1OcTM0QfeIfpxn8SM MdFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from; bh=QReze3KI9/e9+oivGM+ncDQRkYomM2PtVYhGnmXOPSg=; b=0MxC4kWWp5WsUahJzFT8sRsxSN6ENi4ZVdvMHTw5LUY0KHm1ejRrx8pbV2UnEdt4Tx zLtCAYY8Ra1Karx7DqjxTqf6m/CVDT453fuYtrz7AgTJ97rYHYCF2GQY9rMWf2K1DLku xoQWq1c/aJ6ye6fZ6tBT+zwPJO7hZ8Ba8bhmruiPqpT+fzGifEs+mRXix/SoHRSY/G+T 7Vt1KqiXMDixJ68pnu8HE2uyubGQf/osgpJZHchgWcHv2buDj7d8dXte/65/1BPI6hg0 q+0huOqo2vNkL2xJ1W7ZVWle13M9BLcmr4xuA54YXcTQiDnP1iCkfPM/rl3DYAr1PNVc EjmQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com Return-Path: Received: from goliath.siemens.de (goliath.siemens.de. [192.35.17.28]) by gmr-mx.google.com with ESMTPS id s8si427312wme.0.2019.03.26.00.23.18 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 26 Mar 2019 00:23:18 -0700 (PDT) Received-SPF: pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.28 as permitted sender) client-ip=192.35.17.28; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by goliath.siemens.de (8.15.2/8.15.2) with ESMTPS id x2Q7NHMd009971 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 26 Mar 2019 08:23:18 +0100 Received: from ring.ppmd.siemens.net (linux-ses-ext02.ppmd.siemens.net [139.25.69.232]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id x2Q7NHCF004734; Tue, 26 Mar 2019 08:23:17 +0100 From: claudius.heine.ext@siemens.com To: isar-users@googlegroups.com Cc: Claudius Heine Subject: [PATCH v2 4/4] meta/classes/image*: refactor image pipeline and image.bbclass Date: Tue, 26 Mar 2019 08:23:12 +0100 Message-Id: <20190326072312.10863-5-claudius.heine.ext@siemens.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190326072312.10863-1-claudius.heine.ext@siemens.com> References: <20190326072312.10863-1-claudius.heine.ext@siemens.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-TUID: 1NrgWUiASx0Y From: Claudius Heine The new pipeline is roughly as follows: 0. `do_unpack` 1. `do_rootfs` 2. `do_rootfs_postprocess` 3. `do_image_tools` 4. `do_image` 5. `do_deploy` x. `do_build` This commit also splits the image.bbclass functionality up into additional bbclass files to improve independant development. Signed-off-by: Claudius Heine --- meta/classes/ext4-img.bbclass | 2 +- meta/classes/fit-img.bbclass | 2 +- meta/classes/image-cache-extension.bbclass | 26 +++++ meta/classes/image-sdk-extension.bbclass | 36 +++++++ meta/classes/image-tools-extension.bbclass | 43 ++++++++ meta/classes/image.bbclass | 113 +++++---------------- meta/classes/targz-img.bbclass | 2 +- meta/classes/ubi-img.bbclass | 2 +- meta/classes/ubifs-img.bbclass | 2 +- meta/classes/wic-img.bbclass | 4 +- 10 files changed, 136 insertions(+), 96 deletions(-) create mode 100644 meta/classes/image-cache-extension.bbclass create mode 100644 meta/classes/image-sdk-extension.bbclass create mode 100644 meta/classes/image-tools-extension.bbclass diff --git a/meta/classes/ext4-img.bbclass b/meta/classes/ext4-img.bbclass index 5eac507..e2e91a8 100644 --- a/meta/classes/ext4-img.bbclass +++ b/meta/classes/ext4-img.bbclass @@ -24,5 +24,5 @@ do_ext4_image() { _do_ext4_image_cleanup } -addtask ext4_image before do_build after do_copy_boot_files +addtask ext4_image before do_image after do_image_tools do_ext4_image[prefuncs] = 'set_image_size' diff --git a/meta/classes/fit-img.bbclass b/meta/classes/fit-img.bbclass index e8026c4..221ac3f 100644 --- a/meta/classes/fit-img.bbclass +++ b/meta/classes/fit-img.bbclass @@ -24,4 +24,4 @@ do_fit_image() { sudo chroot ${BUILDCHROOT_DIR} /usr/bin/mkimage ${MKIMAGE_ARGS} \ -f '${PP_WORK}/${FIT_IMAGE_SOURCE}' '${PP_DEPLOY}/${FIT_IMAGE_FILE}' } -addtask fit_image before do_build after do_copy_boot_files do_install_imager_deps do_unpack do_transform_template +addtask fit_image before do_image after do_image_tools do_transform_template diff --git a/meta/classes/image-cache-extension.bbclass b/meta/classes/image-cache-extension.bbclass new file mode 100644 index 0000000..af17e0b --- /dev/null +++ b/meta/classes/image-cache-extension.bbclass @@ -0,0 +1,26 @@ +# This software is a part of ISAR. +# Copyright (C) Siemens AG, 2019 +# +# SPDX-License-Identifier: MIT +# +# This class extends the image.bbclass to supply the creation of cache repositories + +inherit base-apt-helper + +do_cache_base_repo[depends] = "base-apt:do_cache_config" +do_cache_base_repo[lockfiles] = "${REPO_BASE_DIR}/isar.lock" +do_cache_base_repo[stamp-extra-info] = "${DISTRO}-${MACHINE}" +do_cache_base_repo() { + if [ -d ${WORKDIR}/apt_cache ]; then + populate_base_apt ${WORKDIR}/apt_cache + fi + + if [ -d ${BUILDCHROOT_HOST_DIR}/var/cache/apt ]; then + populate_base_apt ${BUILDCHROOT_HOST_DIR}/var/cache/apt + fi + + if [ -d ${BUILDCHROOT_TARGET_DIR}/var/cache/apt ]; then + populate_base_apt ${BUILDCHROOT_TARGET_DIR}/var/cache/apt + fi +} +addtask cache_base_repo after do_rootfs do_install_imager_deps diff --git a/meta/classes/image-sdk-extension.bbclass b/meta/classes/image-sdk-extension.bbclass new file mode 100644 index 0000000..1838080 --- /dev/null +++ b/meta/classes/image-sdk-extension.bbclass @@ -0,0 +1,36 @@ +# This software is a part of ISAR. +# Copyright (C) Siemens AG, 2019 +# +# SPDX-License-Identifier: MIT +# +# This class extends the image.bbclass to supply the creation of a sdk + +SDKCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/sdkchroot-${HOST_DISTRO}-${HOST_ARCH}" + +do_populate_sdk[stamp-extra-info] = "${DISTRO}-${MACHINE}" +do_populate_sdk[depends] = "sdkchroot:do_build" +do_populate_sdk() { + # Copy isar-apt with deployed Isar packages + sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} ${SDKCHROOT_DIR}/rootfs/isar-apt + + # Purge apt cache to make image slimmer + sudo rm -rf ${SDKCHROOT_DIR}/rootfs/var/cache/apt/* + + sudo umount -R ${SDKCHROOT_DIR}/rootfs/dev || true + sudo umount ${SDKCHROOT_DIR}/rootfs/proc || true + sudo umount -R ${SDKCHROOT_DIR}/rootfs/sys || true + + # Remove setup scripts + sudo rm -f ${SDKCHROOT_DIR}/rootfs/chroot-setup.sh ${SDKCHROOT_DIR}/rootfs/configscript.sh + + # Copy mount_chroot.sh for convenience + sudo cp ${ISARROOT}/scripts/mount_chroot.sh ${SDKCHROOT_DIR}/rootfs + + # Create SDK archive + sudo tar -C ${SDKCHROOT_DIR} --transform="s|^rootfs|sdk-${DISTRO}-${DISTRO_ARCH}|" \ + -c rootfs | xz -T0 > ${DEPLOY_DIR_IMAGE}/sdk-${DISTRO}-${DISTRO_ARCH}.tar.xz + + # Install deployment link for local use + ln -Tfsr ${SDKCHROOT_DIR}/rootfs ${DEPLOY_DIR_IMAGE}/sdk-${DISTRO}-${DISTRO_ARCH} +} +addtask populate_sdk after do_rootfs diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass new file mode 100644 index 0000000..0a6d5ee --- /dev/null +++ b/meta/classes/image-tools-extension.bbclass @@ -0,0 +1,43 @@ +# This software is a part of ISAR. +# Copyright (C) Siemens AG, 2019 +# +# SPDX-License-Identifier: MIT +# +# This file extends the image.bbclass to supply tools for futher imager functions + +# Imager are expected to run natively, thus will use the target buildchroot. +ISAR_CROSS_COMPILE = "0" + +inherit buildchroot + +IMAGER_INSTALL ??= "" +IMAGER_BUILD_DEPS ??= "" +DEPENDS += "${IMAGER_BUILD_DEPS}" + +python () { + if d.getVar('IMAGE_TYPE', True) == 'wic-img': + d.appendVar('IMAGER_INSTALL', + ' ' + d.getVar('WIC_IMAGER_INSTALL', True)) +} + +do_install_imager_deps[depends] = "buildchroot-target:do_build" +do_install_imager_deps[deptask] = "do_deploy_deb" +do_install_imager_deps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock" +do_install_imager_deps() { + if [ -z "${@d.getVar("IMAGER_INSTALL", True).strip()}" ]; then + exit + fi + + buildchroot_do_mounts + + E="${@bb.utils.export_proxies(d)}" + sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \ + apt-get update \ + -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \ + -o Dir::Etc::sourceparts="-" \ + -o APT::Get::List-Cleanup="0" + apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y \ + --allow-unauthenticated install \ + ${IMAGER_INSTALL}' +} +addtask install_imager_deps before do_image_tools diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index 450b50e..01c6f3b 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -53,8 +53,10 @@ image_do_mounts() { buildchroot_do_mounts } -inherit ${IMAGE_TYPE} inherit isar-bootstrap-helper +inherit image-sdk-extension +inherit image-cache-extension +inherit image-tools-extension # Extra space for rootfs in MB ROOTFS_EXTRA ?= "64" @@ -174,7 +176,6 @@ isar_image_cleanup() { } do_rootfs[depends] = "isar-apt:do_cache_config isar-bootstrap-target:do_bootstrap" - do_rootfs[deptask] = "do_deploy_deb" do_rootfs[root_cleandirs] = "${IMAGE_ROOTFS} \ ${IMAGE_ROOTFS}/isar-apt" @@ -192,9 +193,9 @@ do_mark_rootfs() { --build-id "${BUILD_ID}" --variant "${DESCRIPTION}" \ "${IMAGE_ROOTFS}" } +addtask mark_rootfs before do_rootfs_postprocess after do_rootfs -addtask mark_rootfs before do_copy_boot_files do_transform_template after do_rootfs - +do_copy_boot_files[dirs] = "${DEPLOY_DIR_IMAGE}" do_copy_boot_files() { if [ -n "${KERNEL_IMAGE}" ]; then cp -f ${IMAGE_ROOTFS}/boot/${@get_image_name(d, 'vmlinuz')[0]} ${DEPLOY_DIR_IMAGE}/${KERNEL_IMAGE} @@ -216,97 +217,31 @@ do_copy_boot_files() { cp -f "$dtb" "${DEPLOY_DIR_IMAGE}/${DTB_FILE}" fi } +addtask copy_boot_files before do_rootfs_postprocess after do_rootfs -addtask copy_boot_files before do_build after do_rootfs -do_copy_boot_files[dirs] = "${DEPLOY_DIR_IMAGE}" - -SDKCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/sdkchroot-${HOST_DISTRO}-${HOST_ARCH}" - -do_populate_sdk() { - # Copy isar-apt with deployed Isar packages - sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} ${SDKCHROOT_DIR}/rootfs/isar-apt - - # Purge apt cache to make image slimmer - sudo rm -rf ${SDKCHROOT_DIR}/rootfs/var/cache/apt/* - - sudo umount -R ${SDKCHROOT_DIR}/rootfs/dev || true - sudo umount ${SDKCHROOT_DIR}/rootfs/proc || true - sudo umount -R ${SDKCHROOT_DIR}/rootfs/sys || true - - # Remove setup scripts - sudo rm -f ${SDKCHROOT_DIR}/rootfs/chroot-setup.sh ${SDKCHROOT_DIR}/rootfs/configscript.sh - - # Copy mount_chroot.sh for convenience - sudo cp ${ISARROOT}/scripts/mount_chroot.sh ${SDKCHROOT_DIR}/rootfs - - # Create SDK archive - sudo tar -C ${SDKCHROOT_DIR} --transform="s|^rootfs|sdk-${DISTRO}-${DISTRO_ARCH}|" \ - -c rootfs | xz -T0 > ${DEPLOY_DIR_IMAGE}/sdk-${DISTRO}-${DISTRO_ARCH}.tar.xz - - # Install deployment link for local use - ln -Tfsr ${SDKCHROOT_DIR}/rootfs ${DEPLOY_DIR_IMAGE}/sdk-${DISTRO}-${DISTRO_ARCH} +python do_rootfs_postprocess() { + """Virtual task""" + pass } +addtask rootfs_postprocess before do_build after do_rootfs -do_populate_sdk[depends] = "sdkchroot:do_build" - -addtask populate_sdk after do_rootfs - -inherit base-apt-helper - -do_cache_base_repo[depends] = "base-apt:do_cache_config" -do_cache_base_repo[lockfiles] = "${REPO_BASE_DIR}/isar.lock" - -do_cache_base_repo() { - if [ -d ${WORKDIR}/apt_cache ]; then - populate_base_apt ${WORKDIR}/apt_cache - fi - - if [ -d ${BUILDCHROOT_HOST_DIR}/var/cache/apt ]; then - populate_base_apt ${BUILDCHROOT_HOST_DIR}/var/cache/apt - fi - - if [ -d ${BUILDCHROOT_TARGET_DIR}/var/cache/apt ]; then - populate_base_apt ${BUILDCHROOT_TARGET_DIR}/var/cache/apt - fi +python do_image_tools() { + """Virtual task""" + pass } +addtask image_tools before do_build after do_rootfs_postprocess -addtask cache_base_repo after do_rootfs do_install_imager_deps - -# Imager are expected to run natively, thus will use the target buildchroot. -ISAR_CROSS_COMPILE = "0" - -inherit buildchroot - -IMAGER_INSTALL ??= "" -IMAGER_BUILD_DEPS ??= "" -DEPENDS += "${IMAGER_BUILD_DEPS}" - -python () { - if d.getVar('IMAGE_TYPE', True) == 'wic-img': - d.appendVar('IMAGER_INSTALL', - ' ' + d.getVar('WIC_IMAGER_INSTALL', True)) +python do_image() { + """Virtual task""" + pass } +addtask image before do_build after do_image_tools -do_install_imager_deps() { - if [ -z "${@d.getVar("IMAGER_INSTALL", True).strip()}" ]; then - exit - fi - - buildchroot_do_mounts - - E="${@bb.utils.export_proxies(d)}" - sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \ - apt-get update \ - -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \ - -o Dir::Etc::sourceparts="-" \ - -o APT::Get::List-Cleanup="0" - apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y \ - --allow-unauthenticated install \ - ${IMAGER_INSTALL}' +python do_deploy() { + """Virtual task""" + pass } +addtask deploy before do_build after do_image -do_install_imager_deps[depends] = "buildchroot-target:do_build" -do_install_imager_deps[deptask] = "do_deploy_deb" -do_install_imager_deps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock" - -addtask install_imager_deps before do_build +# Last so that the image type can overwrite tasks if needed +inherit ${IMAGE_TYPE} diff --git a/meta/classes/targz-img.bbclass b/meta/classes/targz-img.bbclass index 0a0ece7..f408527 100644 --- a/meta/classes/targz-img.bbclass +++ b/meta/classes/targz-img.bbclass @@ -10,4 +10,4 @@ do_targz_image() { sudo tar -cvzf ${TARGZ_IMAGE_FILE} -C ${IMAGE_ROOTFS} . } -addtask targz_image before do_build after do_mark_rootfs +addtask targz_image before do_image after do_imager_tools diff --git a/meta/classes/ubi-img.bbclass b/meta/classes/ubi-img.bbclass index a34c465..2cf3344 100644 --- a/meta/classes/ubi-img.bbclass +++ b/meta/classes/ubi-img.bbclass @@ -27,4 +27,4 @@ do_ubi_image() { sudo chroot ${BUILDCHROOT_DIR} /usr/sbin/ubinize ${UBINIZE_ARGS} \ -o '${PP_DEPLOY}/${UBI_IMAGE_FILE}' '${PP_WORK}/${UBINIZE_CFG}' } -addtask ubi_image before do_build after do_copy_boot_files do_install_imager_deps do_unpack do_transform_template +addtask ubi_image before do_image after do_image_tools do_transform_template diff --git a/meta/classes/ubifs-img.bbclass b/meta/classes/ubifs-img.bbclass index bf019a8..4297d21 100644 --- a/meta/classes/ubifs-img.bbclass +++ b/meta/classes/ubifs-img.bbclass @@ -23,4 +23,4 @@ do_ubifs_image() { -r '${PP_ROOTFS}' '${PP_DEPLOY}/${UBIFS_IMAGE_FILE}' } -addtask ubifs_image before do_build after do_copy_boot_files do_install_imager_deps +addtask ubifs_image before do_image after do_image_tools diff --git a/meta/classes/wic-img.bbclass b/meta/classes/wic-img.bbclass index 3ca8607..74368a6 100644 --- a/meta/classes/wic-img.bbclass +++ b/meta/classes/wic-img.bbclass @@ -133,7 +133,7 @@ python do_rootfs_wicenv () { } -addtask do_rootfs_wicenv after do_copy_boot_files before do_wic_image +addtask do_rootfs_wicenv after do_rootfs_postprocess before do_wic_image do_rootfs_wicenv[vardeps] += "${WICVARS}" do_rootfs_wicenv[prefuncs] = 'set_image_size' @@ -167,4 +167,4 @@ EOSUDO do_wic_image[file-checksums] += "${WKS_FILE_CHECKSUM}" do_wic_image[depends] = "buildchroot-target:do_build" -addtask wic_image before do_build after do_install_imager_deps +addtask wic_image before do_image after do_image_tools -- 2.20.1