Re: [PATCH v2 RESEND 2/2] delay creation of initrd until end of rootfs install

[Uladzimir Bely <ubely@ilbers.de>]

On Tue, 2024-09-17 at 11:15 +0200, 'Felix Moessbauer' via isar-users
wrote:
> This patch solves major performance issues around the initramfs
> creation by ensuring that the initrd is only created once. This is
> implemented by stubbing the update-initramfs call during the package
> installing. After all apt operations are completed, we manually
> trigger the initrd creation. In case a custom initramfs is used, the
> creation is completely skipped in the image rootfs, as this would
> anyways not be used.
> 
> Before that, each package install that made a initrd relevant change
> triggered the update of the initrd. As we have multiple apt calls
> during
> the build, this step was sometimes executed multiple times. In
> addition,
> the apt install step is emulated, further slowing down the initrd
> generation. On some layers on non native architecutes, this summed up
> to
> over 10 minutes of initrd generation time.
> 
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
>  meta/classes/image.bbclass                    |  4 +++
>  meta/classes/rootfs.bbclass                   | 32
> +++++++++++++++++++
>  .../isar-bootstrap/isar-bootstrap.inc         |  2 ++
>  3 files changed, 38 insertions(+)
> 
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 0dea06fa..b6763d73 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -67,6 +67,8 @@ inherit essential
>  
>  ROOTFSDIR = "${IMAGE_ROOTFS}"
>  ROOTFS_FEATURES += "clean-package-cache clean-pycache generate-
> manifest export-dpkg-status clean-log-files clean-debconf-cache"
> +# when using a custom initrd, do not generate one as part of the
> image rootfs
> +ROOTFS_FEATURES += "${@ '' if d.getVar('INITRD_IMAGE') == '' else
> 'no-generate-initrd'}"
>  ROOTFS_PACKAGES += "${IMAGE_PREINSTALL}
> ${@isar_multiarch_packages('IMAGE_INSTALL', d)}"
>  ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}"
>  ROOTFS_DPKGSTATUS_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}"
> @@ -458,6 +460,8 @@ do_rootfs_quality_check() {
>      args="${ROOTFS_QA_FIND_ARGS}"
>      # rootfs_finalize chroot-setup.sh
>      args="${args} ! -path ${ROOTFSDIR}/var/lib/dpkg/diversions"
> +    # initramfs is generated outside of the image rootfs
> +    args="${args} ! -path ${ROOTFSDIR}/boot/initrd.img*"

Hello.

This fails in CI (at least with focal/arm64 target with

DEBUG: Executing shell function do_rootfs_quality_check
find: paths must precede expression: `<path_to>/build/tmp/work/ubuntu-
focal-arm64/isar-image-base-qemuarm64/1.0-
r0/rootfs/boot/initrd.img.old'
WARNING: exit code 1 from a shell command.

>      for cmd in ${ROOTFS_POSTPROCESS_COMMAND}; do
>          case "${cmd}" in
>  	    image_postprocess_mark)
> diff --git a/meta/classes/rootfs.bbclass
> b/meta/classes/rootfs.bbclass
> index f0abd795..5c7a4c9a 100644
> --- a/meta/classes/rootfs.bbclass
> +++ b/meta/classes/rootfs.bbclass
> @@ -14,6 +14,7 @@ ROOTFS_BASE_DISTRO ?= "${BASE_DISTRO}"
>  # 'generate-manifest' - generate a package manifest of the rootfs
> into ${ROOTFS_MANIFEST_DEPLOY_DIR}
>  # 'export-dpkg-status' - exports /var/lib/dpkg/status file to
> ${ROOTFS_DPKGSTATUS_DEPLOY_DIR}
>  # 'clean-log-files' - delete log files that are not owned by
> packages
> +# 'no-generate-initrd' - do not generate debian default initrd
>  ROOTFS_FEATURES ?= ""
>  
>  ROOTFS_APT_ARGS="install --yes -o Debug::pkgProblemResolver=yes"
> @@ -127,6 +128,16 @@ rootfs_configure_apt() {
>  EOSUDO
>  }
>  
> +ROOTFS_CONFIGURE_COMMAND += "rootfs_disable_initrd_generation"
> +rootfs_disable_initrd_generation[weight] = "1"
> +rootfs_disable_initrd_generation() {
> +    # fully disable initrd generation
> +    echo "replace update-initramfs with stub"
> +    sudo mv "${ROOTFSDIR}/usr/sbin/update-initramfs" \
> +            "${ROOTFSDIR}/usr/sbin/update-initramfs.isar"
> +    sudo chroot "${ROOTFSDIR}" ln -s "/usr/bin/true"
> "/usr/sbin/update-initramfs"
> +}
> +
>  
>  ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_update"
>  rootfs_install_pkgs_update[weight] = "5"
> @@ -331,6 +342,27 @@ rootfs_cleanup_base_apt() {
>  EOSUDO
>  }
>  
> +ROOTFS_POSTPROCESS_COMMAND += "rootfs_restore_initrd_tooling"
> +rootfs_restore_initrd_tooling[weight] = "1"
> +rootfs_restore_initrd_tooling() {
> +    if [ -e "${ROOTFSDIR}/usr/sbin/update-initramfs.isar" ]; then
> +        sudo mv -f "${ROOTFSDIR}/usr/sbin/update-initramfs.isar" \
> +            "${ROOTFSDIR}/usr/sbin/update-initramfs"
> +    fi
> +}
> +
> +ROOTFS_POSTPROCESS_COMMAND +=
> "${@bb.utils.contains('ROOTFS_FEATURES', 'no-generate-initrd', '',
> 'rootfs_generate_initrd', d)}"
> +rootfs_generate_initrd[weight] = "10"
> +rootfs_generate_initrd() {
> +    if [ -n "$(sudo find '${ROOTFSDIR}/boot' -type f -name
> 'vmlinu[xz]*')" ]; then
> +        sudo -E chroot "${ROOTFSDIR}" sh -c '\
> +            export kernel_version=$(basename /boot/vmlinu[xz]* | cut
> -d'-' -f2-); \
> +            update-initramfs -u -v -k "$kernel_version";'
> +    else
> +        echo "no kernel in this rootfs, do not generate initrd"
> +    fi
> +}
> +
>  do_rootfs_postprocess[vardeps] = "${ROOTFS_POSTPROCESS_COMMAND}"
>  do_rootfs_postprocess[network] = "${TASK_USE_SUDO}"
>  python do_rootfs_postprocess() {
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> index 12f32ff0..e85c2894 100644
> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> @@ -28,6 +28,8 @@ DISTRO_BOOTSTRAP_KEYS ?= ""
>  THIRD_PARTY_APT_KEYS ?= ""
>  DEPLOY_ISAR_BOOTSTRAP ?= ""
>  DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"
> +# install early, so we can stub the update-initramfs script before
> rootfs install
> +DISTRO_BOOTSTRAP_BASE_PACKAGES:append = ",initramfs-tools"
>  DISTRO_BOOTSTRAP_BASE_PACKAGES:append:gnupg = ",gnupg"
>  DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = ",ca-
> certificates"
>  DISTRO_VARS_PREFIX ?= "${@'HOST_' if
> bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else ''}"
> -- 
> 2.39.2
> 

-- 
Best regards,
Uladzimir.

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/badf50c25c78318d37f29dbcda7fbf73dc060c72.camel%40ilbers.de.