From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7184619192186830848 X-Received: by 2002:a05:651c:1583:b0:277:e8e:8d90 with SMTP id h3-20020a05651c158300b002770e8e8d90mr3733324ljq.243.1672799510678; Tue, 03 Jan 2023 18:31:50 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:ac2:4e8a:0:b0:4c8:8384:83f3 with SMTP id o10-20020ac24e8a000000b004c8838483f3ls2339226lfr.3.-pod-prod-gmail; Tue, 03 Jan 2023 18:31:49 -0800 (PST) X-Google-Smtp-Source: AMrXdXsNPRa/eiRgM2xvnQHMxQm0ATr9wqwcwGVBRDoBWbkPevBn6UqnqXRHvAEKMNDvDrNNpm79 X-Received: by 2002:ac2:4e91:0:b0:4cb:13d7:77e2 with SMTP id o17-20020ac24e91000000b004cb13d777e2mr7471772lfr.26.1672799508985; Tue, 03 Jan 2023 18:31:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672799508; cv=none; d=google.com; s=arc-20160816; b=w1+6z0IkbYKnvfTfv6GVp4iJcR93Zg7M03KPtESoHDh2FeF46XnkZcCchzr8TN5/HG 5AwIUqaA64eDTUyenXAbWK2ARLjhaJiYS3/Qm6BIqXzqhbUzSaI7g5JasQXSkK8rUseh 3RK/b49tMa5eAi2ydJ4oRkOfa/mBEBX8kN7iC+VXZ1zcfRYhfTr1/SMCMOeOQ/dlejnK tPzFGnDFJrS24EbF1WyLvwQ7JQ3ogqMAWSuRFDNiN5PAgWTXgtCyoNBnhxv3yu7a9pMa FddIzcJKEsyD9YrGltEbhFwjRPoLZ2RWrxN0Z2Mtd80ZXgHP+lC5oZRvEXINbZGaPdMi Ln6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:message-id:date :subject:cc:to:from:dkim-signature; bh=T+cbW1zlne3EyTPOL+tpjTR/l6FPNPjItvN82boZUXQ=; b=ZqIZTECZycJTUwoSOkQwtBZLuibYoonzp8MTaE7kOfXbTXlkcU82gQtu737geyvMvX JuB+ZQi1CPRVcf63x9EX1rSw/dhx9E9rzvAYv5gnphO7wMmtvUye13KQZ5GGu2asYanm 4VJSORzUukpV0zAvc/V7YePno3sl46adHFl4OVK4rFtq3De/seBGI31dejWAFETnLcnN z9qSs7rv4JCk2x/epIBL2Vub+qHl1qVqDPXRzjNQs0IxAji3cBaDA7oV92KvIRWR8sPm ogleNuLLFmV2XIl6nhWfWVJjzp3JulSzg6hkijj1WdLJE2wPqNf7vpz/KSDBfKGtaNaB NUzA== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=FIlwvcU2; spf=pass (google.com: domain of fm-72506-20230104023147c9675e33bcfa82736c-_0yg1t@rts-flowmailer.siemens.com designates 185.136.64.227 as permitted sender) smtp.mailfrom=fm-72506-20230104023147c9675e33bcfa82736c-_0yG1t@rts-flowmailer.siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from mta-64-227.siemens.flowmailer.net (mta-64-227.siemens.flowmailer.net. [185.136.64.227]) by gmr-mx.google.com with ESMTPS id s4-20020a056512202400b004abdb5d1128si1141377lfs.2.2023.01.03.18.31.48 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 03 Jan 2023 18:31:48 -0800 (PST) Received-SPF: pass (google.com: domain of fm-72506-20230104023147c9675e33bcfa82736c-_0yg1t@rts-flowmailer.siemens.com designates 185.136.64.227 as permitted sender) client-ip=185.136.64.227; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=FIlwvcU2; spf=pass (google.com: domain of fm-72506-20230104023147c9675e33bcfa82736c-_0yg1t@rts-flowmailer.siemens.com designates 185.136.64.227 as permitted sender) smtp.mailfrom=fm-72506-20230104023147c9675e33bcfa82736c-_0yG1t@rts-flowmailer.siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: by mta-64-227.siemens.flowmailer.net with ESMTPSA id 20230104023147c9675e33bcfa82736c for ; Wed, 04 Jan 2023 03:31:47 +0100 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=fm1; d=siemens.com; i=felix.moessbauer@siemens.com; h=Date:From:Subject:To:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:Cc; bh=T+cbW1zlne3EyTPOL+tpjTR/l6FPNPjItvN82boZUXQ=; b=FIlwvcU2D4wSLaCH6OUW8wA9EtCy4gcBcfEWftbzpy1WrPGVtccrmibS1EUHtx9+ooWsF1 8t61OOjzHLivGzwxuoKkNcISlxqldheZyp90XPLE9LPTtaQ6AdRhRZCBpW5LFBk2pgj32ttr JC6D8riQOcZmOYkWIMo/4nEqFPzLU=; From: Felix Moessbauer To: isar-users@googlegroups.com Cc: florian.bezdeka@siemens.com, roberto.foglietta@gmail.com, ubely@ilbers.de, henning.schild@siemens.com, jan.kiszka@siemens.com, Felix Moessbauer Subject: [PATCH 1/1] Reduce disk usage during build by avoiding copies of apt cache Date: Wed, 4 Jan 2023 02:31:28 +0000 Message-Id: <20230104023128.374390-1-felix.moessbauer@siemens.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-72506:519-21489:flowmailer X-TUID: Ce7BjkqcLHC1 From: "Roberto A. Foglietta" This patch leverages few ways to accomplish the task to optimise the large buildings. It uses hard physical links instead of copying files which is a faster way and saves disk spaces. Put a CACHEDIR.TAG file in each cache folder in such a way every tar that uses the option --exclude-caches could be avoided to include debian packages saving a lot of time and disk space. Signed-off-by: Roberto A. Foglietta Signed-off-by: Felix Moessbauer --- This is a cleaned up and squashed version of the following commits from Robertos ISAR fork: - 83ce56580c39a2f708f9a3ede73439c82edac1be - 02bfafb20a8c43585561b9de306a2e23c5ff50bb In addition, I applied some small fixes to fix issues on rebuild. This is a compromise between the large patch series of Roberto and the too minimalistic one of Uladzimir. As the series from Roberto is not on this list, I want to send out this patch to have something which we can review. Maybe we can integrate that first to solve 90% of the issue and then continue improving it. Best regards, Felix meta/classes/deb-dl-dir.bbclass | 11 ++++++----- meta/classes/dpkg.bbclass | 6 +++--- meta/classes/rootfs.bbclass | 8 ++++---- meta/classes/sdk.bbclass | 2 +- meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 3 ++- 5 files changed, 16 insertions(+), 14 deletions(-) diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass index 3b1517d..3a87390 100644 --- a/meta/classes/deb-dl-dir.bbclass +++ b/meta/classes/deb-dl-dir.bbclass @@ -84,9 +84,9 @@ deb_dl_dir_import() { flock -s "${pc}".lock -c ' set -e printenv | grep -q BB_VERBOSE_LOGS && set -x - + sudo touch "${rootfs}"/var/cache/apt/archives/CACHEDIR.TAG sudo find "${pc}" -type f -iname "*\.deb" -exec \ - cp -n --no-preserve=owner -t "${rootfs}"/var/cache/apt/archives/ {} + + ln -Pf -t "${rootfs}"/var/cache/apt/archives/ {} + 2>/dev/null || : ' } @@ -97,20 +97,21 @@ deb_dl_dir_export() { flock "${pc}".lock -c ' set -e printenv | grep -q BB_VERBOSE_LOGS && set -x - + sudo touch "${rootfs}"/var/cache/apt/archives/CACHEDIR.TAG find "${rootfs}"/var/cache/apt/archives/ \ -maxdepth 1 -type f -iname '*\.deb' |\ while read p; do # skip files from a previous export - [ -f "${pc}/${p##*/}" ] && continue + [ -e "${pc}/${p##*/}" ] && continue # can not reuse bitbake function here, this is basically # "repo_contains_package" package=$(find "${REPO_ISAR_DIR}"/"${DISTRO}" -name ${p##*/}) if [ -n "$package" ]; then cmp --silent "$package" "$p" && continue fi - sudo cp -n "${p}" "${pc}" + sudo ln -P "${p}" "${pc}" 2>/dev/null || : done + sudo touch "${pc}/CACHEDIR.TAG" sudo chown -R $(id -u):$(id -g) "${pc}" ' } diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass index 7822b14..166ec16 100644 --- a/meta/classes/dpkg.bbclass +++ b/meta/classes/dpkg.bbclass @@ -108,10 +108,10 @@ dpkg_runbuild() { --chroot-setup-commands="echo \"Package: *\nPin: release n=${DEBDISTRONAME}\nPin-Priority: 1000\" > /etc/apt/preferences.d/isar-apt" \ --chroot-setup-commands="echo \"APT::Get::allow-downgrades 1;\" > /etc/apt/apt.conf.d/50isar-apt" \ --chroot-setup-commands="rm -f /var/log/dpkg.log" \ - --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \ + --chroot-setup-commands="ln -Pf ${ext_deb_dir}/*.deb -t ${deb_dir}/ 2>/dev/null || :" \ --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \ - --finished-build-commands="cp -n --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \ - --finished-build-commands="cp /var/log/dpkg.log ${ext_root}/dpkg_partial.log" \ + --finished-build-commands="ln -P ${deb_dir}/*.deb -t ${ext_deb_dir}/ 2>/dev/null || :" \ + --finished-build-commands="cp /var/log/dpkg.log ${ext_root}/dpkg_partial.log; touch ${ext_deb_dir}/CACHEDIR.TAG" \ --debbuildopts="--source-option=-I" \ --build-dir=${WORKDIR} --dist="isar" ${DSC_FILE} diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index 786682d..147f517 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -78,7 +78,7 @@ BOOTSTRAP_SRC_${ROOTFS_ARCH} = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTF rootfs_prepare[weight] = "25" rootfs_prepare(){ - sudo cp -Trpfx '${BOOTSTRAP_SRC}/' '${ROOTFSDIR}' + sudo cp -Trpfx --reflink=auto '${BOOTSTRAP_SRC}/' '${ROOTFSDIR}' } ROOTFS_CONFIGURE_COMMAND += "rootfs_configure_isar_apt" @@ -115,7 +115,6 @@ rootfs_configure_apt() { EOSUDO } - ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_update" rootfs_install_pkgs_update[weight] = "5" rootfs_install_pkgs_update[isar-apt-lock] = "acquire-before" @@ -218,7 +217,7 @@ cache_deb_src() { # Note: ISAR updates the apt state information(apt-get update) only once during bootstrap and # relies on that through out the build. Copy that state information instead of apt-get update # which generates a new state from upstream. - sudo cp -Trpn "${BOOTSTRAP_SRC}/var/lib/apt/lists/" "${ROOTFSDIR}/var/lib/apt/lists/" + sudo cp -Trpn --reflink=auto "${BOOTSTRAP_SRC}/var/lib/apt/lists/" "${ROOTFSDIR}/var/lib/apt/lists/" deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME} debsrc_download ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME} @@ -308,7 +307,8 @@ rootfs_install_sstate_prepare() { # so we use some mount magic to prevent that mkdir -p ${WORKDIR}/mnt/rootfs sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro - sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar --one-file-system rootfs + lopts="--one-file-system --exclude-caches --exclude=var/cache/apt/archives" + sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar $lopts rootfs sudo umount ${WORKDIR}/mnt/rootfs sudo chown $(id -u):$(id -g) rootfs.tar } diff --git a/meta/classes/sdk.bbclass b/meta/classes/sdk.bbclass index 79e01a1..3feada2 100644 --- a/meta/classes/sdk.bbclass +++ b/meta/classes/sdk.bbclass @@ -82,7 +82,7 @@ do_rootfs_install[vardeps] += "${SDKROOTFSVARDEPS}" ROOTFS_CONFIGURE_COMMAND_append_class-sdk = " ${@'rootfs_configure_isar_apt_dir' if d.getVar('SDK_INCLUDE_ISAR_APT') == '1' else ''}" rootfs_configure_isar_apt_dir() { # Copy isar-apt instead of mounting: - sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} ${ROOTFSDIR}/isar-apt + sudo cp -Trpfx --reflink=auto ${REPO_ISAR_DIR}/${DISTRO} ${ROOTFSDIR}/isar-apt } ROOTFS_POSTPROCESS_COMMAND_prepend_class-sdk = "sdkchroot_configscript " diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index b9ae16c..d53db4d 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -418,7 +418,8 @@ SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize" bootstrap_sstate_prepare() { # this runs in SSTATE_BUILDDIR, which will be deleted automatically - sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar --one-file-system $(basename "${ROOTFSDIR}") + lopts="--one-file-system --exclude-caches --exclude=var/cache/apt/archives" + sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar $lopts $(basename "${ROOTFSDIR}") sudo chown $(id -u):$(id -g) bootstrap.tar } -- 2.34.1