From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7184619192186830848 X-Received: by 2002:a1f:c345:0:b0:3b8:2d76:434d with SMTP id t66-20020a1fc345000000b003b82d76434dmr4966492vkf.31.1672803428622; Tue, 03 Jan 2023 19:37:08 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a1f:988a:0:b0:3b4:4e41:3c89 with SMTP id a132-20020a1f988a000000b003b44e413c89ls4233615vke.6.-pod-prod-gmail; Tue, 03 Jan 2023 19:37:07 -0800 (PST) X-Google-Smtp-Source: AMrXdXvCfQq2T8d4awXzujC/ns2oTp6qUlqeyJMhoJSW8P+WhF5+tL/hES51ahi2gL/1yy+qyKPQ X-Received: by 2002:a1f:ec02:0:b0:3d5:91de:1d1e with SMTP id k2-20020a1fec02000000b003d591de1d1emr8834162vkh.3.1672803427797; Tue, 03 Jan 2023 19:37:07 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672803427; cv=none; d=google.com; s=arc-20160816; b=pDghV87lEyJukrpmiX0/QF3fzR2ynXedn/7AyiIGcbIPPnnJ6LyEzyE4q9gIfX4vVy gz/beXYTDAtcZYzL2S0fNtnbU9oc4ziS9gNAxn3lOAdHyiCmKvhSPggin6znRQ+vV+KC 2Wnsat+1GiGD7ZlZhqQQKQmNJxAUtEgKLYD9yyQt/SvzCrxZ2Pe3EkDt9D2+oSIig2iG EK887gkRwLZHkdH1RwWxDsmGufzvEkXxLUs+bf2R718se2ksHdHR3PCss1O2mnOmuytn t5cd5nZ4u9u0q5RqBvp6a8lD54Lh/QP9kpm4Fca9oxoVX5pStvp62Rx9UuJIJ43HglqG u+/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:dkim-signature; bh=kVTxjn/uW2mDIsUnn7e399tdH+6udhpIS2sEl+xDxAI=; b=bWuuP/9HYCFskB3AHMTxuBPBcX1pPsio03Vv+YsSEHgtDDAB93nqp+rgVfivI/5cdY qRCKfl9thlLFhegE1kYIDhMCXUZOChjO4jRpjuKYAA8AyzHkoW43BsB2FhTXdFpXArBv Ktg/Vbjc/8KzSb40yyAlH2htJxiOpIB1LRpNZueyXpZO5NWZfjdNa1Fjl8O+Dd+1SQV+ Qz/eJvjiRZYZ/AhoNkcKziwN2D/cc+gwrIEHzGMKPXgdTpdVKNt5DNDkiIqPaD3lc6ca xqvzxGV16eIampQMdHLX1R5c9XPyhBMdzSkxs3ifNoSmq0cAfQVdb3KVVTA7mENlej+S j5fg== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=bN59LuId; spf=pass (google.com: domain of roberto.foglietta@gmail.com designates 2607:f8b0:4864:20::f2e as permitted sender) smtp.mailfrom=roberto.foglietta@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail-qv1-xf2e.google.com (mail-qv1-xf2e.google.com. [2607:f8b0:4864:20::f2e]) by gmr-mx.google.com with ESMTPS id y80-20020a1f3253000000b003d53e3ed270si2302297vky.0.2023.01.03.19.37.07 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 03 Jan 2023 19:37:07 -0800 (PST) Received-SPF: pass (google.com: domain of roberto.foglietta@gmail.com designates 2607:f8b0:4864:20::f2e as permitted sender) client-ip=2607:f8b0:4864:20::f2e; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=bN59LuId; spf=pass (google.com: domain of roberto.foglietta@gmail.com designates 2607:f8b0:4864:20::f2e as permitted sender) smtp.mailfrom=roberto.foglietta@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: by mail-qv1-xf2e.google.com with SMTP id i12so22847084qvs.2 for ; Tue, 03 Jan 2023 19:37:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=kVTxjn/uW2mDIsUnn7e399tdH+6udhpIS2sEl+xDxAI=; b=bN59LuIdbTPLjGsPfs/68P1KrEL5XiJPjNENkUEYUItlIwg+fZjN+snmaK25b/noJS m0KaQFwGk/ijjLeNNpja/UCXlx4gJrY/ScixmV/LJRU6ukioqRIfZca6XM8qRo0eAc8+ lE71woYw//PlDYpKF8KAJ20tZIRfk7Xtv0nXj0Rp49HHDhHJSbCX0A7cs/cSWZyphEl2 B0il0mHAiPCbaaZmecpUDU424AhCksmVSQuS3Y/71zAZgeO/aYrVXqLAOmpPM8jxgqLu z5bbCiwNKziO0YFn5oBCNjn8+0RZk9IBtXa7GO6SAYgT7GTVD3gwVPeaVfrXTH2kmT9u GsuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=kVTxjn/uW2mDIsUnn7e399tdH+6udhpIS2sEl+xDxAI=; b=3ZglExO9UrJAau2qDbXWdTZraKj1W1Q18mUcOsAjl8DvQnRRAqS3MQN59mZ11qpUtY YY/cngLJRRktP8PEHD3w8VfmSn93ZsSW2HHwDTxuDg/5g6Mii05SBKfhbdsWpvlWitiN qZRBMxmicqzwutCqnKlpcufnFB8GjCrPtWR2CX5OZsb9eTrA7ILeDsa/oXnRGbEgyz/f uMRkdX2xaaexaSPUj18qUV4xNs5BF/DfPd1dUSHYGdvstwW5gbalhDCAGHedwZRZYKLT Be0xPABTgCWk08uqrSaybC5nxXo1Im85enTHSio3ensji8jNXCUQ911zj91X4jRmzAd5 WDwA== X-Gm-Message-State: AFqh2kqj+vo/zmZi446xAaMH7d4lxQBPJoL8DHCwJfvHOpYjaXhkLhCX iufSRpCxS+lXQtwpwBB7urbSIOp8nSL6a9t1y5vW9LYG1w== X-Received: by 2002:a0c:eb92:0:b0:4d8:8f4e:3a3b with SMTP id x18-20020a0ceb92000000b004d88f4e3a3bmr2360462qvo.16.1672803427362; Tue, 03 Jan 2023 19:37:07 -0800 (PST) MIME-Version: 1.0 References: <20230104023128.374390-1-felix.moessbauer@siemens.com> In-Reply-To: <20230104023128.374390-1-felix.moessbauer@siemens.com> From: "Roberto A. Foglietta" Date: Wed, 4 Jan 2023 04:36:30 +0100 Message-ID: Subject: Re: [PATCH 1/1] Reduce disk usage during build by avoiding copies of apt cache To: Felix Moessbauer Cc: isar-users@googlegroups.com, florian.bezdeka@siemens.com, ubely@ilbers.de, henning.schild@siemens.com, jan.kiszka@siemens.com Content-Type: text/plain; charset="UTF-8" X-TUID: IlWkZ5Vb8I47 On Wed, 4 Jan 2023 at 03:31, Felix Moessbauer wrote: > > From: "Roberto A. Foglietta" > > This patch leverages few ways to accomplish the task to optimise the large > buildings. It uses hard physical links instead of copying files which is a > faster way and saves disk spaces. Put a CACHEDIR.TAG file in each cache folder > in such a way every tar that uses the option --exclude-caches could be avoided > to include debian packages saving a lot of time and disk space. Hi Felix, The CACHEDIR is not the best but the following that uses --sxclude= + sudo tar --one-file-system --exclude="var/log/*" --exclude="var/cache/*" --exclude="var/backups/*" \ + --exclude="var/tmp/*" --exclude="var/crash/*" --exclude="var/spool/*" --exclude="var/lib/apt/*" \ + --exclude-caches --exclude-backups -C ${WORKDIR}/mnt -cpSf rootfs.tar rootfs > In addition, I applied some small fixes to fix issues on rebuild. The fixes are not in this patch. > - > + sudo touch "${rootfs}"/var/cache/apt/archives/CACHEDIR.TAG > find "${rootfs}"/var/cache/apt/archives/ \ > -maxdepth 1 -type f -iname '*\.deb' |\ > while read p; do > # skip files from a previous export > - [ -f "${pc}/${p##*/}" ] && continue > + [ -e "${pc}/${p##*/}" ] && continue > # can not reuse bitbake function here, this is basically > # "repo_contains_package" > package=$(find "${REPO_ISAR_DIR}"/"${DISTRO}" -name ${p##*/}) > if [ -n "$package" ]; then > cmp --silent "$package" "$p" && continue > fi This code is broken and even if it would work with /bin/dash, it is useless. Moreover this function does not support the transfer of data to minimize the time of apt-get update. Another huge change is due to writing the .wic image (folder) directly on the destination + the patches about schroot Everything on my NEXT branch is valuable to be integrated == FASTER AND DISK SAVING == * ee44d4e - deb_dl_dir_im/export nolists when USE_CCACHE is not active * 2ad229b - sstate cache not anymore CACHEDIR.TAG but tar --exclude * c442591 - changes for a faster build using less disk space, p.6 * 1e69b8f - changes for a faster build using less disk space, p.3 (v2) * 997809a - changes for a faster build using less disk space, p.2 * fee43dc - changes for a faster build using less disk space, p.1 == BUG FIXES & Co. == * 6415204 - image tools ext. class defines ISAR_CROSS_COMPILE, if missing * 06916be - dpkg class set default value for ISAR_CROSS_COMPILE, if missing == FLEXIBILITY == * 7e72921 - dpkg class sbuild allows extra arguments by vars == MIGRATION TO SCHROOT == * 79e97dd - dpkg base class: use schroot only not buildchroot anymore * 662383a - events: Cleanup lost schroot sessions if any, v2 * 9750bd5 - imager: Move image types to schroot, v2 * 51b08db - imager: Migrate from buildchroot to schroot, v2 * 0fea6b1 - sbuild: Allow setting custom config paths, v2 I did the cherry picking before answering Felix, so the patchset is not tested on next yet. https://github.com/ilbers/isar/compare/next...robang74:isar:next Best regards, R-