From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Tue, 17 Sep 2024 11:15:28 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-pl1-f191.google.com (mail-pl1-f191.google.com [209.85.214.191]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 48H9FQiX004616 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 17 Sep 2024 11:15:27 +0200 Received: by mail-pl1-f191.google.com with SMTP id d9443c01a7336-205443cf91fsf55918585ad.1 for ; Tue, 17 Sep 2024 02:15:27 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1726564520; cv=pass; d=google.com; s=arc-20240605; b=G1d7vCVPgq6CZFI94sQSME1jkdLIpha0Vady95LUUubfr61e9Ed6Pf7o95dvY1wbwk 8cr5bI9r5XaDfEAUMwAYxYHNM4VHuanUdw9n2oVRNqEkDYQylFqgdRS3r1WUAl4BpoyK lz7kfyh5UqcVgxLxxccgb9rwcKiGOmL5zuzExnkv6eb6HmLIg7dfa7MpWXWKoX88PMw9 clm4vJyKxJ9INY6yiY8TmZPjl9f6Cbj/QPnEulyVo77+e+K6oTh4Qzs1+d8w2qQFol1E BlPZwOAJJDJQWTzOpcXcuLdAafsdsXtbawWU0a6kkNMfnL1MbgAWT1MKAhv7lzrrTbWS SNbg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=tpXm8MNvgqwc/Qa1Kal6iW7SgGgzKDlhGZV4UHGE8yw=; fh=bsnbhYhObRn8bUg/nleO5bi9avUcQEi5289+bf6IqWo=; b=dMrola8ZUeChZzDXWNJM5wUqHvBvddEabcY/5AK/SBG6CiaSaY3D136z8ohpWZkZIm s0Dw35+2Y9t7zL3SeGp53ehmkCVN43pCZBKpROmM80OFCtqG+RotYE2aCzz4zxUvy/S1 YzjvgSX6OLvlw1eR8B0v5ea5Dvy0JtoohrQ55HTeo537e++DzR4OcJxdBKsDEx+2eCiM ELpyFh1RUql8vzHo45EcauNLIWvvKNUvXDZjMzpf9kDPVjMc3LrXOWqfCeOjVbyiH5vQ I2keQY+uW88OeS5YgohgV0khcqsnYTErhSTnUkhsFr8nObUybPBYpAWqlOdkIuI1S9H1 ijxQ==; darn=ilbers.de ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=RS+j7SCX; spf=pass (google.com: domain of fm-1321639-2024091709151633d5fbc78a73267740-xiujew@rts-flowmailer.siemens.com designates 185.136.64.227 as permitted sender) smtp.mailfrom=fm-1321639-2024091709151633d5fbc78a73267740-XiUJew@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1726564520; x=1727169320; darn=ilbers.de; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=tpXm8MNvgqwc/Qa1Kal6iW7SgGgzKDlhGZV4UHGE8yw=; b=pUsetJ3rms+5MD4HqNp0J2R7BBpGbXRHA9B1zMevaMs7DvAqUNhjIgTC7sSfUsNFIq 17DFM/Pv0HN7Tk3cG679lLemg2W5ew3i7QOj8WTyDd8uzK1Iqm6K1uDwBKw7zPJU/597 LE6lMm2g/UvIylfS4UIofM1N9OaMOFCLw09iPs+9XzC7sEoqy++9JrMI7kOqOCoJmDmc 6vruEzVRpXbI8D2+fQ9xBt5xKjGTVxI1iaUmvb39qABCFQdxqcMJ8gLQx4THmxkwbv/t FMhTpGncryxJ3ay0unrvOeFfJ/zKpH18rvwXWGjEr2gqWAHJL6stiaLXIpWk9/Z3x1aC TaIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726564520; x=1727169320; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=tpXm8MNvgqwc/Qa1Kal6iW7SgGgzKDlhGZV4UHGE8yw=; b=kat8xinxgH5iq7vCGZgkkxnlaZb78LTiC9dFVkC1Ke6KAvmb8E6jfkDBobFsffEgjR 9F7rJXZY8Gh2G+DzaNyBdM02TLqojMFUYewE3Z98hPyU0SwAzgJuYq8sOjiSbF1OvtEW nN1j/KJISPijU6qL+XMGEVG8DLgqAsT7wrkMyvgzTmQ0JZZ7ZQxpDjguKwXEn18j9085 UojrEDfiX19wuSlTAvAxeRbGikGPSWhhjKMiY+/ggXc/Yz5tZ89/P+jBvHBC4YjYL8AP K0ipPbuYcvlpmKUD0N2K0sZPqf5hSaCadAw0br4xDHKcNSwhnCDd/Ocbh6LV6/RFPY6a 7r5Q== X-Forwarded-Encrypted: i=2; AJvYcCVUUWRN/H9E3Jct1ftdXgMcnSFMQLhN0zbDs1EVTiBcUQ7QnV+wVlpPQHNkWDbQPrTm9Z5j@ilbers.de X-Gm-Message-State: AOJu0Yz+fZ8ntaGv3x4rUPab+BbywDk1fRWQev1AvV3o/D/bhgee3TyT pEtxlPGI/1ki5zfWnp6QauuVAiFlW3OJ2oM/7YweMulkSh8hG8Re X-Google-Smtp-Source: AGHT+IGCv0JjJdeY2vCfTYUWi81v2ajJV4syE+dqVqVqIC8ExI2OTDSI6G+pjQzyWpC7k4J0uUd2Ig== X-Received: by 2002:a17:902:e88a:b0:205:8a25:904b with SMTP id d9443c01a7336-2076e45c40emr273152445ad.57.1726564520063; Tue, 17 Sep 2024 02:15:20 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a17:903:32cf:b0:206:9820:463c with SMTP id d9443c01a7336-2076cafaf19ls10470475ad.1.-pod-prod-01-us; Tue, 17 Sep 2024 02:15:19 -0700 (PDT) X-Received: by 2002:a17:902:d50e:b0:207:ded:9cbd with SMTP id d9443c01a7336-2076e370cadmr263841565ad.24.1726564518722; Tue, 17 Sep 2024 02:15:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1726564518; cv=none; d=google.com; s=arc-20240605; b=hPygGS81oiLx5mOFY58YK0jwsE5PDCnUOu1bUkVS0+4Z2xkeELUFp7S/rtZowWIwRm eo+zriUZnATg7pMGFVmAsnsNMryTbUuwhL6ntGPR77xqGzGpMFXm3vO+QqelktM4KZzc lYlHX5/cJFEjOdBPi72yegBeCKo1dogFNAPBF8HI30KHQ0vsVSrdanKcC7b1LNgf5Wny IEZ6qq2G5CWCp4O7ecikm+wKSjtzIdt9PLW8rjtidACqRjg+HowlWAnF8kpmSRRDuB4X 9e0pDJfD6kAB2NGHvN7sFiExsfdnJLJwKLLs82RI1wocKhlEMrn7rOYM97XyuRyMiirn uRdw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=PgQFQrj4bMhchxQMFG0fCBitlAX781ezjnzBRu3bjms=; fh=2vPEtVRFZgJExMSCfX53nKBFsjiPYp/60rbGy5S5Mwo=; b=UWoYR7+WL6PEI1WFXH0VsAFkpaIRUz2jfMY4TR0HBYAVR3xky8iPlbi1pkTBwQBE+S ltFyOmXvhuL2CrK5pkV/LG6VpTB/Cqg2RH+FKPCNrSzmnSbL4aEY5wS/7/yjRoSAeq01 DWtJEJLxmvkUSk34SyQI18HamkVzNU2i7yml0htJWu0NBQ7O2PnLq9L+JmBXtCay/F+u WQ7oCVTrLt3VwIuO9dGem9KDYu3TDXPeY9HNl+DOPBUgVYUJLC81BYFfCWs4lbS/8jl9 3tg8F9yYKYDI5G/ZrlChUisCjc/xaMnGMxfIMLukV1JTiybRv3nOrWc9Q3Ktq2i+hait 7MPw==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=RS+j7SCX; spf=pass (google.com: domain of fm-1321639-2024091709151633d5fbc78a73267740-xiujew@rts-flowmailer.siemens.com designates 185.136.64.227 as permitted sender) smtp.mailfrom=fm-1321639-2024091709151633d5fbc78a73267740-XiUJew@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-64-227.siemens.flowmailer.net (mta-64-227.siemens.flowmailer.net. [185.136.64.227]) by gmr-mx.google.com with ESMTPS id d9443c01a7336-207946c2f60si2770715ad.8.2024.09.17.02.15.18 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 17 Sep 2024 02:15:18 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-1321639-2024091709151633d5fbc78a73267740-xiujew@rts-flowmailer.siemens.com designates 185.136.64.227 as permitted sender) client-ip=185.136.64.227; Received: by mta-64-227.siemens.flowmailer.net with ESMTPSA id 2024091709151633d5fbc78a73267740 for ; Tue, 17 Sep 2024 11:15:16 +0200 From: "'Felix Moessbauer' via isar-users" To: isar-users@googlegroups.com Cc: jan.kiszka@siemens.com, Felix Moessbauer Subject: [PATCH v2 RESEND 2/2] delay creation of initrd until end of rootfs install Date: Tue, 17 Sep 2024 11:15:07 +0200 Message-Id: <20240917091507.1714450-2-felix.moessbauer@siemens.com> In-Reply-To: <20240917091507.1714450-1-felix.moessbauer@siemens.com> References: <20240917091507.1714450-1-felix.moessbauer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1321639:519-21489:flowmailer X-Original-Sender: felix.moessbauer@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=RS+j7SCX; spf=pass (google.com: domain of fm-1321639-2024091709151633d5fbc78a73267740-xiujew@rts-flowmailer.siemens.com designates 185.136.64.227 as permitted sender) smtp.mailfrom=fm-1321639-2024091709151633d5fbc78a73267740-XiUJew@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Felix Moessbauer Reply-To: Felix Moessbauer Content-Type: text/plain; charset="UTF-8" Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: gYOhdObPvd0m This patch solves major performance issues around the initramfs creation by ensuring that the initrd is only created once. This is implemented by stubbing the update-initramfs call during the package installing. After all apt operations are completed, we manually trigger the initrd creation. In case a custom initramfs is used, the creation is completely skipped in the image rootfs, as this would anyways not be used. Before that, each package install that made a initrd relevant change triggered the update of the initrd. As we have multiple apt calls during the build, this step was sometimes executed multiple times. In addition, the apt install step is emulated, further slowing down the initrd generation. On some layers on non native architecutes, this summed up to over 10 minutes of initrd generation time. Signed-off-by: Felix Moessbauer --- meta/classes/image.bbclass | 4 +++ meta/classes/rootfs.bbclass | 32 +++++++++++++++++++ .../isar-bootstrap/isar-bootstrap.inc | 2 ++ 3 files changed, 38 insertions(+) diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index 0dea06fa..b6763d73 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -67,6 +67,8 @@ inherit essential ROOTFSDIR = "${IMAGE_ROOTFS}" ROOTFS_FEATURES += "clean-package-cache clean-pycache generate-manifest export-dpkg-status clean-log-files clean-debconf-cache" +# when using a custom initrd, do not generate one as part of the image rootfs +ROOTFS_FEATURES += "${@ '' if d.getVar('INITRD_IMAGE') == '' else 'no-generate-initrd'}" ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${@isar_multiarch_packages('IMAGE_INSTALL', d)}" ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}" ROOTFS_DPKGSTATUS_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}" @@ -458,6 +460,8 @@ do_rootfs_quality_check() { args="${ROOTFS_QA_FIND_ARGS}" # rootfs_finalize chroot-setup.sh args="${args} ! -path ${ROOTFSDIR}/var/lib/dpkg/diversions" + # initramfs is generated outside of the image rootfs + args="${args} ! -path ${ROOTFSDIR}/boot/initrd.img*" for cmd in ${ROOTFS_POSTPROCESS_COMMAND}; do case "${cmd}" in image_postprocess_mark) diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index f0abd795..5c7a4c9a 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -14,6 +14,7 @@ ROOTFS_BASE_DISTRO ?= "${BASE_DISTRO}" # 'generate-manifest' - generate a package manifest of the rootfs into ${ROOTFS_MANIFEST_DEPLOY_DIR} # 'export-dpkg-status' - exports /var/lib/dpkg/status file to ${ROOTFS_DPKGSTATUS_DEPLOY_DIR} # 'clean-log-files' - delete log files that are not owned by packages +# 'no-generate-initrd' - do not generate debian default initrd ROOTFS_FEATURES ?= "" ROOTFS_APT_ARGS="install --yes -o Debug::pkgProblemResolver=yes" @@ -127,6 +128,16 @@ rootfs_configure_apt() { EOSUDO } +ROOTFS_CONFIGURE_COMMAND += "rootfs_disable_initrd_generation" +rootfs_disable_initrd_generation[weight] = "1" +rootfs_disable_initrd_generation() { + # fully disable initrd generation + echo "replace update-initramfs with stub" + sudo mv "${ROOTFSDIR}/usr/sbin/update-initramfs" \ + "${ROOTFSDIR}/usr/sbin/update-initramfs.isar" + sudo chroot "${ROOTFSDIR}" ln -s "/usr/bin/true" "/usr/sbin/update-initramfs" +} + ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_update" rootfs_install_pkgs_update[weight] = "5" @@ -331,6 +342,27 @@ rootfs_cleanup_base_apt() { EOSUDO } +ROOTFS_POSTPROCESS_COMMAND += "rootfs_restore_initrd_tooling" +rootfs_restore_initrd_tooling[weight] = "1" +rootfs_restore_initrd_tooling() { + if [ -e "${ROOTFSDIR}/usr/sbin/update-initramfs.isar" ]; then + sudo mv -f "${ROOTFSDIR}/usr/sbin/update-initramfs.isar" \ + "${ROOTFSDIR}/usr/sbin/update-initramfs" + fi +} + +ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'no-generate-initrd', '', 'rootfs_generate_initrd', d)}" +rootfs_generate_initrd[weight] = "10" +rootfs_generate_initrd() { + if [ -n "$(sudo find '${ROOTFSDIR}/boot' -type f -name 'vmlinu[xz]*')" ]; then + sudo -E chroot "${ROOTFSDIR}" sh -c '\ + export kernel_version=$(basename /boot/vmlinu[xz]* | cut -d'-' -f2-); \ + update-initramfs -u -v -k "$kernel_version";' + else + echo "no kernel in this rootfs, do not generate initrd" + fi +} + do_rootfs_postprocess[vardeps] = "${ROOTFS_POSTPROCESS_COMMAND}" do_rootfs_postprocess[network] = "${TASK_USE_SUDO}" python do_rootfs_postprocess() { diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index 12f32ff0..e85c2894 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -28,6 +28,8 @@ DISTRO_BOOTSTRAP_KEYS ?= "" THIRD_PARTY_APT_KEYS ?= "" DEPLOY_ISAR_BOOTSTRAP ?= "" DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales" +# install early, so we can stub the update-initramfs script before rootfs install +DISTRO_BOOTSTRAP_BASE_PACKAGES:append = ",initramfs-tools" DISTRO_BOOTSTRAP_BASE_PACKAGES:append:gnupg = ",gnupg" DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = ",ca-certificates" DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else ''}" -- 2.39.2 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240917091507.1714450-2-felix.moessbauer%40siemens.com.