From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 19 Sep 2024 08:56:56 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f187.google.com (mail-lj1-f187.google.com [209.85.208.187]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 48J6usZn014784 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 19 Sep 2024 08:56:55 +0200 Received: by mail-lj1-f187.google.com with SMTP id 38308e7fff4ca-2f77ca80fe1sf4309651fa.0 for ; Wed, 18 Sep 2024 23:56:55 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1726729009; cv=pass; d=google.com; s=arc-20240605; b=F+IiobAsQJRJUVftxMjoKlkXOXb15rPTr5CbsQoF39mjYylRU+mFXdDTVHb8Klqxuu /ASorUVvyONbWksmI4uW1go51QFxFfUJlePNzdcMg3qmrWjQpaqHfwDVgjyqj9bwgoPv K7wwLDlIeAzmsU4JFYKXPTuhWd7ezH6KowQVUXLWpDKbjxsOKKveV6TXugBLNgKdXRCw PNBWE3S0C3g3WGqdnaFyqAtVh+tu8IKRG1ry+M3xrJSnTvLW8CgoOSxSgfLw6/nXv3JD 1GdQl3vQ7Wgk/8nIOb7j0JhZo9zobaqHsj4MDTaByzpgwLUGF+ABerm4ch5I0ypdpJXB G/lQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:user-agent :content-transfer-encoding:references:in-reply-to:date:to:from :subject:message-id:sender:dkim-signature; bh=+us9uOwGod91g0aFdusdf6kH32cCzGozVpq1b7M4V78=; fh=3WFtqYvK/fvlF+rpopXtjMg4PV7Ze6Tw7sY2nd6F/R4=; b=VRX+h3bqRsuXNrRyUt/X3gyNkW/iJIa3sJrzWjPdEhvVx0CWNsHlkMDitXGWPmuE+G w3QoaXLGQc6ZLzn6Zcts+gZkLvJOyHn+pOqDSGI3uNfOrnD8lyutNc2qlpmNWDEL97BM Tha+D/3v7ewKUuPEXWdmjH8pilCIxwKRsVxY3bmbRDz3EF6StlXRnWeDyrZFcozVBRAF JviUtpMqkikMIvXTVgJAiJCqaL9L7G/dov0WNV/8rh/iv37oyYdo48B4oRw+Ru2VubHO dBjfS+bTxV8uI2aq/qzZjk5u09eO8yCK+hLdeQPV3o5/OvFZD3zxvxMyiwoCH2Z85o3L Pifg==; darn=ilbers.de ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1726729009; x=1727333809; darn=ilbers.de; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:user-agent:content-transfer-encoding :references:in-reply-to:date:to:from:subject:message-id:sender:from :to:cc:subject:date:message-id:reply-to; bh=+us9uOwGod91g0aFdusdf6kH32cCzGozVpq1b7M4V78=; b=Uq2jHHNyLTIpdHYUy/08sB6V4L+Uu3/PHKaPfpeZ6kuAaqMiNUCjiq0PvJbtwnJaDJ S5Bk+eaxDkOEPOnyXDADp8n3pd5slIUPPqQLMcLXGY42Er+1lPcwkDk4pcweLYWRTt/6 PWZNUA6jtuPTqNkGcUc0F+vsOBXiLkZa52kNV6QDNN/H2sA0UR1Pa4PRBx2JtIj4MTN7 DH50ddpkXWaRbSeiXOS0/+8F8YLtvgJbaDi3BoiFQDUvwItdp+QjhliSTUcJGz9m9Pwc 0oDmDZG6sHOLBJX3MBi2W6RiXvM45AsSkMTDQ6RkZNkkGnM9K6IFzpdHtwdCWfHSZMWc OpGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726729009; x=1727333809; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :user-agent:content-transfer-encoding:references:in-reply-to:date:to :from:subject:message-id:x-beenthere:x-gm-message-state:sender:from :to:cc:subject:date:message-id:reply-to; bh=+us9uOwGod91g0aFdusdf6kH32cCzGozVpq1b7M4V78=; b=rOdF9PKT7x9a8XxAY4yfj6eFIbVducty8DStrLKx/XI/+rk0tm8emb1TWpUR0rFxPu 8702j8R0riKm55MMSio1gioFyx0odeYcwcy4KeVygBeQ26pNNz/8n3+a4yfZzjCMS0Ov FZA91EoQWoTnrEz5hy7T8/9N6lt8u9+fbUpy9oAH6J1CbT7HC8D32gS85hz1R1L+aNP0 7ogidyNMlVYlSW1uBcBFajm9vBFzu1Rdhk0XbFY1Uku7VwRnkgnZ+yHPb6ux0tTIGUc8 9/cYH3e9ZQirEwZuMfl3bx1O0bCwXlsP+O7RR4C+/FjPNB9xjt5ZR3c8oK3ym3VulExO 4P8g== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCWe6o7EXtSMmEm+RgD12RLM2ldEBKaaTabxXJ1dIAqfYnpoYp1ZN5Wt+cBge/eu1EJWdyiK@ilbers.de X-Gm-Message-State: AOJu0Yzfyjxq6WtSPqEOlYMp4xZWxc8QABYoL6A73DFwaLWXUzK/i8Iz ogJNoF4GWOk01n7KTd/bFKrJ1ba3iNwH64THBpfXuMqcx1phWoL3 X-Google-Smtp-Source: AGHT+IGxa1kWG8piz5UQRo4XPZfax//OKN2uje7/760nVPHbfVO8Q40A6mLnAQ3gkt9A8DSvc65SAw== X-Received: by 2002:a05:651c:547:b0:2f5:a29:5a42 with SMTP id 38308e7fff4ca-2f787dc3dc3mr145301091fa.14.1726729008302; Wed, 18 Sep 2024 23:56:48 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:2205:0:b0:2f3:f0f9:1a32 with SMTP id 38308e7fff4ca-2f7c3e04c3fls2240341fa.0.-pod-prod-03-eu; Wed, 18 Sep 2024 23:56:46 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCXtpfHePI6TinzhNtGIIASdCvFiHlVZqvRdKy4/8sYYXZQn1xxdUSU10o/itb34NW8HD+mx9eZpcsCU@googlegroups.com X-Received: by 2002:a05:651c:2205:b0:2f4:f472:c4fa with SMTP id 38308e7fff4ca-2f787f3ba7dmr149850401fa.35.1726729005699; Wed, 18 Sep 2024 23:56:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1726729005; cv=none; d=google.com; s=arc-20240605; b=OoWxf/TMQZ6Nf/zx8VSNC1XVfbxYOfP5ChWI4KDxcJqR7BcvnWgiZgXlp5Pq3tFASU XZ8pGJ0EeMkbrkVk98QghXB1jwMp6ycOsspRDuS7YNaTwt5mztwhv6vlsd6mpT2fok+U mDsQxw+3d2l20NMIrES78l9dSaXWvzfEzU/8BZIZkuwunts8g+XIVFNHlEwytwjyYN5t FEprLOx8sXxe1kubArMsdUrtBUaUoRUBGJXMxhhxCELoYL0e6HmKXvG1KcFCf6w0rTjl kiTKD0JtHJ8SVHcOlXqrhxYGqT/16ar8btTLwh2lpxFvTTYI0sRLa4NaYUQACK8TNQYy eccw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:to:from:subject:message-id; bh=Kh9cZ4x6zi5UIiOYu8nmhO2Zdvkjhbj6ScSA1YNemM4=; fh=ez+yYIN2+t1/A1KtVD1essDQ6+anTTnZTbSXjuK9WaU=; b=V8BBhOO8ttuNhBSG7q9fk7Wg6UP2zYvyvntNhh826wZW6TajRdWYQ/DK/LJhXOcXV2 zj7EZXyKwFBFxGH4MmP8o4KPnRa/VsK+GsmbMNtxXFCK5ao5nhYzNAdbjLTe4av7Vnjy TMQQAJQIXLUfdlgRQEQE5KSolGGZzfnbAbkJYifvKagy0Kie9CpzYojTw67HIIuhTicn YM2R8SK75c2MQQf202pKP6WssnmyAk/5c/POSRJnV8wJ9UCr8AAuuaWUB8LmDhMVeWZy ritC7j7Aw2N2jmJEj1vHsjGGbdTg4WWG8uDSsxtV5cxxi9Mt0/2uHSwvLgn4vSzkyPEk Dvww==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-42e75465a87si269255e9.1.2024.09.18.23.56.45 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Wed, 18 Sep 2024 23:56:45 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from [127.0.0.1] (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 48J6ugvR014777 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 19 Sep 2024 08:56:44 +0200 Message-ID: Subject: Re: [PATCH v2 RESEND 2/2] delay creation of initrd until end of rootfs install From: Uladzimir Bely To: Felix Moessbauer , isar-users@googlegroups.com Date: Thu, 19 Sep 2024 09:56:42 +0300 In-Reply-To: <20240917091507.1714450-2-felix.moessbauer@siemens.com> References: <20240917091507.1714450-1-felix.moessbauer@siemens.com> <20240917091507.1714450-2-felix.moessbauer@siemens.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.52.4 (by Flathub.org) MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-TUID: krJvWhp/efQP On Tue, 2024-09-17 at 11:15 +0200, 'Felix Moessbauer' via isar-users wrote: > This patch solves major performance issues around the initramfs > creation by ensuring that the initrd is only created once. This is > implemented by stubbing the update-initramfs call during the package > installing. After all apt operations are completed, we manually > trigger the initrd creation. In case a custom initramfs is used, the > creation is completely skipped in the image rootfs, as this would > anyways not be used. >=20 > Before that, each package install that made a initrd relevant change > triggered the update of the initrd. As we have multiple apt calls > during > the build, this step was sometimes executed multiple times. In > addition, > the apt install step is emulated, further slowing down the initrd > generation. On some layers on non native architecutes, this summed up > to > over 10 minutes of initrd generation time. >=20 > Signed-off-by: Felix Moessbauer > --- > =C2=A0meta/classes/image.bbclass=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= |=C2=A0 4 +++ > =C2=A0meta/classes/rootfs.bbclass=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 | 32 > +++++++++++++++++++ > =C2=A0.../isar-bootstrap/isar-bootstrap.inc=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 |=C2=A0 2 ++ > =C2=A03 files changed, 38 insertions(+) >=20 > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index 0dea06fa..b6763d73 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -67,6 +67,8 @@ inherit essential > =C2=A0 > =C2=A0ROOTFSDIR =3D "${IMAGE_ROOTFS}" > =C2=A0ROOTFS_FEATURES +=3D "clean-package-cache clean-pycache generate- > manifest export-dpkg-status clean-log-files clean-debconf-cache" > +# when using a custom initrd, do not generate one as part of the > image rootfs > +ROOTFS_FEATURES +=3D "${@ '' if d.getVar('INITRD_IMAGE') =3D=3D '' else > 'no-generate-initrd'}" > =C2=A0ROOTFS_PACKAGES +=3D "${IMAGE_PREINSTALL} > ${@isar_multiarch_packages('IMAGE_INSTALL', d)}" > =C2=A0ROOTFS_MANIFEST_DEPLOY_DIR ?=3D "${DEPLOY_DIR_IMAGE}" > =C2=A0ROOTFS_DPKGSTATUS_DEPLOY_DIR ?=3D "${DEPLOY_DIR_IMAGE}" > @@ -458,6 +460,8 @@ do_rootfs_quality_check() { > =C2=A0=C2=A0=C2=A0=C2=A0 args=3D"${ROOTFS_QA_FIND_ARGS}" > =C2=A0=C2=A0=C2=A0=C2=A0 # rootfs_finalize chroot-setup.sh > =C2=A0=C2=A0=C2=A0=C2=A0 args=3D"${args} ! -path ${ROOTFSDIR}/var/lib/dpk= g/diversions" > +=C2=A0=C2=A0=C2=A0 # initramfs is generated outside of the image rootfs > +=C2=A0=C2=A0=C2=A0 args=3D"${args} ! -path ${ROOTFSDIR}/boot/initrd.img*= " Hello. This fails in CI (at least with focal/arm64 target with DEBUG: Executing shell function do_rootfs_quality_check find: paths must precede expression: `/build/tmp/work/ubuntu- focal-arm64/isar-image-base-qemuarm64/1.0- r0/rootfs/boot/initrd.img.old' WARNING: exit code 1 from a shell command. > =C2=A0=C2=A0=C2=A0=C2=A0 for cmd in ${ROOTFS_POSTPROCESS_COMMAND}; do > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 case "${cmd}" in > =C2=A0 =C2=A0=C2=A0=C2=A0 image_postprocess_mark) > diff --git a/meta/classes/rootfs.bbclass > b/meta/classes/rootfs.bbclass > index f0abd795..5c7a4c9a 100644 > --- a/meta/classes/rootfs.bbclass > +++ b/meta/classes/rootfs.bbclass > @@ -14,6 +14,7 @@ ROOTFS_BASE_DISTRO ?=3D "${BASE_DISTRO}" > =C2=A0# 'generate-manifest' - generate a package manifest of the rootfs > into ${ROOTFS_MANIFEST_DEPLOY_DIR} > =C2=A0# 'export-dpkg-status' - exports /var/lib/dpkg/status file to > ${ROOTFS_DPKGSTATUS_DEPLOY_DIR} > =C2=A0# 'clean-log-files' - delete log files that are not owned by > packages > +# 'no-generate-initrd' - do not generate debian default initrd > =C2=A0ROOTFS_FEATURES ?=3D "" > =C2=A0 > =C2=A0ROOTFS_APT_ARGS=3D"install --yes -o Debug::pkgProblemResolver=3Dyes= " > @@ -127,6 +128,16 @@ rootfs_configure_apt() { > =C2=A0EOSUDO > =C2=A0} > =C2=A0 > +ROOTFS_CONFIGURE_COMMAND +=3D "rootfs_disable_initrd_generation" > +rootfs_disable_initrd_generation[weight] =3D "1" > +rootfs_disable_initrd_generation() { > +=C2=A0=C2=A0=C2=A0 # fully disable initrd generation > +=C2=A0=C2=A0=C2=A0 echo "replace update-initramfs with stub" > +=C2=A0=C2=A0=C2=A0 sudo mv "${ROOTFSDIR}/usr/sbin/update-initramfs" \ > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "${RO= OTFSDIR}/usr/sbin/update-initramfs.isar" > +=C2=A0=C2=A0=C2=A0 sudo chroot "${ROOTFSDIR}" ln -s "/usr/bin/true" > "/usr/sbin/update-initramfs" > +} > + > =C2=A0 > =C2=A0ROOTFS_INSTALL_COMMAND +=3D "rootfs_install_pkgs_update" > =C2=A0rootfs_install_pkgs_update[weight] =3D "5" > @@ -331,6 +342,27 @@ rootfs_cleanup_base_apt() { > =C2=A0EOSUDO > =C2=A0} > =C2=A0 > +ROOTFS_POSTPROCESS_COMMAND +=3D "rootfs_restore_initrd_tooling" > +rootfs_restore_initrd_tooling[weight] =3D "1" > +rootfs_restore_initrd_tooling() { > +=C2=A0=C2=A0=C2=A0 if [ -e "${ROOTFSDIR}/usr/sbin/update-initramfs.isar"= ]; then > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sudo mv -f "${ROOTFSDIR}/usr/= sbin/update-initramfs.isar" \ > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "${RO= OTFSDIR}/usr/sbin/update-initramfs" > +=C2=A0=C2=A0=C2=A0 fi > +} > + > +ROOTFS_POSTPROCESS_COMMAND +=3D > "${@bb.utils.contains('ROOTFS_FEATURES', 'no-generate-initrd', '', > 'rootfs_generate_initrd', d)}" > +rootfs_generate_initrd[weight] =3D "10" > +rootfs_generate_initrd() { > +=C2=A0=C2=A0=C2=A0 if [ -n "$(sudo find '${ROOTFSDIR}/boot' -type f -nam= e > 'vmlinu[xz]*')" ]; then > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sudo -E chroot "${ROOTFSDIR}"= sh -c '\ > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 expor= t kernel_version=3D$(basename /boot/vmlinu[xz]* | cut > -d'-' -f2-); \ > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 updat= e-initramfs -u -v -k "$kernel_version";' > +=C2=A0=C2=A0=C2=A0 else > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 echo "no kernel in this rootf= s, do not generate initrd" > +=C2=A0=C2=A0=C2=A0 fi > +} > + > =C2=A0do_rootfs_postprocess[vardeps] =3D "${ROOTFS_POSTPROCESS_COMMAND}" > =C2=A0do_rootfs_postprocess[network] =3D "${TASK_USE_SUDO}" > =C2=A0python do_rootfs_postprocess() { > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > index 12f32ff0..e85c2894 100644 > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > @@ -28,6 +28,8 @@ DISTRO_BOOTSTRAP_KEYS ?=3D "" > =C2=A0THIRD_PARTY_APT_KEYS ?=3D "" > =C2=A0DEPLOY_ISAR_BOOTSTRAP ?=3D "" > =C2=A0DISTRO_BOOTSTRAP_BASE_PACKAGES =3D "locales" > +# install early, so we can stub the update-initramfs script before > rootfs install > +DISTRO_BOOTSTRAP_BASE_PACKAGES:append =3D ",initramfs-tools" > =C2=A0DISTRO_BOOTSTRAP_BASE_PACKAGES:append:gnupg =3D ",gnupg" > =C2=A0DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support =3D ",ca- > certificates" > =C2=A0DISTRO_VARS_PREFIX ?=3D "${@'HOST_' if > bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else ''}" > --=20 > 2.39.2 >=20 --=20 Best regards, Uladzimir. --=20 You received this message because you are subscribed to the Google Groups "= isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an e= mail to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/= isar-users/badf50c25c78318d37f29dbcda7fbf73dc060c72.camel%40ilbers.de.